A Disentangled Adversarial Neural Topic Model for Separating Opinions from Plots in User Reviews

The flexibility of the inference process in Variational Autoencoders (VAEs) has recently led to revising traditional probabilistic topic models giving rise to Neural Topic Models (NTM). Although these approaches have achieved significant results, surprisingly very little work has been done on how to disentangle the latent topics. Existing topic models when applied to reviews may extract topics associated with writers’ subjective opinions mixed with those related to factual descriptions such as plot summaries in movie and book reviews. It is thus desirable to automatically separate opinion topics from plot/neutral ones enabling a better interpretability. In this paper, we propose a neural topic model combined with adversarial training to disentangle opinion topics from plot and neutral ones. We conduct an extensive experimental assessment introducing a new collection of movie and book reviews paired with their plots, namely MOBO dataset, showing an improved coherence and variety of topics, a consistent disentanglement rate, and sentiment classification performance superior to other supervised topic models.


Introduction
Variational Autoencoders (VAEs) (Kingma and Welling, 2014) allow to design complex generative models of data. In the wake of the renewed interest for VAEs, traditional probabilistic topic models (Blei et al., 2003) have been revised giving rise to several Neural Topic Model (NTM) variants, such as NVDM (Miao et al., 2016), ProdLDA (Srivastava and Sutton, 2017), NTM-R (Ding et al., 2018), etc. However, existing topic models when applied to user reviews may extract topics associated with writers' subjective opinions mixed with those related to factual descriptions such as plot summaries of movies and books (Lin et al., 2012). Although

Disentangled Topics
Positive Plot Negative Figure 1: Disentangled topics extracted by DI-ATOM from the Amazon reviews for "The Hobbit".
these approaches have achieved significant results via the neural inference process, surprisingly very little work has been done on how to disentangle the inferred topic representations. Disentangled representations can be defined as representations where individual latent units are sensitive to variations of a single generative factor, while being relatively invariant to changes of other factors (Bengio et al., 2013;Higgins et al., 2017). Inducing such representations has been shown to be significantly beneficial for their generalization and interpretability (Achille and Soatto, 2018;Peng et al., 2019). For example, an image can be viewed as the results of several generative factors mutually interacting, as one or many sources of light, the material and reflective properties of various surfaces or the shape of the objects depicted (Bengio et al., 2013). In the context of topic modeling, documents result from a generative process over mixtures of latent topics, and therefore, we propose to consider these latent topics as generative factors to be disentangled to improve their interpretability and discriminative power. Disentangled topics are topics invariant to the factors of variation of text, which for instance, in the context of book and movie reviews could be the author's opinion (e.g. positive/negative), the salient parts of a plot or other auxiliary information reported. An illustration of this is shown in Fig. 1 in which opinion topics are separated from plot topics.
However, models relying solely on sentiment information are easily misled and not suitable to disentangle opinion from plots, since even plot descriptions frequently make large use of sentiment expressions (Pang and Lee, 2004a). Consider for example the following sentence: "The ring holds a dark power, and it soon begins to exert its evil influence on Bilbo", an excerpt from a strong positive Amazon's review.
Therefore, we propose to distinguish opinionbearing topics from plot/neutral ones combining a neural topic model architecture with an adversarial training. In this study, we present the DIsentangled Adversarial TOpic Model (DIATOM) 1 , aiming at disentangling information related to the target labels (i.e. the review score), from other distinct aspects yet possibly still polarised (e.g. plot descriptions). We also introduce a new dataset, namely the MOBO dataset 1 , made up of movie and book reviews, paired with their related plots. The reviews come from different publicly available datasets: IMDB (Maas et al., 2011), GoodReads (Wan et al., 2019) and Amazon reviews (McAuley et al., 2015), and encompass a wide spectrum of domains and styles. We conduct an extensive experimental assessment of our model. First, we assess the topic quality in terms of topic coherence and diversity and compare DIATOM with other supervised topic models on the sentiment classification task; then, we analyse the disentangling rate of topics to quantitatively assess the degree of separation between actual opinion and plot/neutral topics.
Our contributions are summarized below: • We propose a new model, DIATOM, which is able to generate disentangled topics through the combination of VAE and adversarial learning.
• We introduce the MOBO dataset, a new collection of movie and book reviews paired with their plots.
• We conduct an experimental assessment of our model, highlighting more interpretable topics with better topic coherence and diversity scores compared to others state-of-the-art supervised topic models, and improved discriminative power on sentiment classification, 1 Source code and dataset omitted for the anonymous submission. and a consistent topic-disentanglement rate.
The rest of the paper is organized as follows. We review the related literature on sentiment-topic models, neural topic models and the studies on disentangled representations ( §2). Then, we present the details of our proposed DIATOM model ( §3), followed by the experimental setup ( §4) and results ( §5). Finally, we conclude with a summary of the results and suggestions for future works ( §6).

Related Work
Our work is closely related to three lines of research: sentiment-topic models, neural topic models and learning disentangled representations. Sentiment-Topic Models. Probabilistic graphical models for topic extraction have been extensively studied. Beyond LDA (Blei et al., 2003); a wide spectrum of models has extended LDA to more specific tasks using contextual information (Rosen-Zvi et al., 2004;Wang and McCallum, 2006;Blei and Lafferty, 2006). Supervised-LDA (sLDA) (Mcauliffe and Blei, 2008) is a generalpurpose supervised extension which builds on top of LDA by adding a response variable associated with each document (e.g. a review's rating).
A category of extensions particularly relevant for this work is the sentiment-topic models. Examples include the Joint Sentiment-Topic (JST) model (Lin and He, 2009;Lin et al., 2012) and Aspect and Sentiment Unification Model (ASUM) (Jo and Oh, 2011). These models are able to extract informative topics grouped under different sentiment classes. Although they do not rely on document labels, they require word prior polarity information to be incorporated into the learning process in order to generate consistent results. Nevertheless, when provided with document-level class labels, JST can learn document-topic distributions influenced by the class information. The possibility to supervise the learning process with document labels and to avoid the necessity of prior information over words makes it suitable for a fair comparison with the model proposed in this work. Besides, these models require carefully tailored inference algorithms, and the standard Gibbs sampling algorithm used can have a high computational cost when fitting large-scale data, with time and memory scaling linearly with the number of documents, leading researchers to devise more sophisticated approaches to make it scalable (Gonzalez et al., 2011).
Compared to DIATOM, the discussed sentiment topic models can only distinguish between polarity-  bearing topics and neutral ones, remaining strictly aligned to the provided labels. Instead, DIATOM is able to generate opinion-bearing topics and plot topics which may still be polarized but not carrying any user's opinion. Neural Topic Models. Neural models provide a more generic and extendable alternative to topic modeling, and therefore, have recently gained increasing interest. Some of them use belief networks (Mnih and Gregor, 2014), or enforce the Dirichlet prior on the document-topic distribution by means of Wasserstein Autoencoders (Nan et al., 2019). Others adopt continuous representations to capture long-term dependencies or preserve word order via sequence-to-sequence VAE (Dieng et al., 2017;Xu et al., 2017;Bowman et al., 2016;Yang et al., 2017) whose time complexity and difficulty of training, however, have limited their applications. Neural Variational Document Model (NVDM) (Miao et al., 2016) is a direct extension of VAE used for topic detection in text. In NVDM, the prior of latent topics is assumed to be a Gaussian distribution. This is not ideal since it cannot mimic the simplex in the latent topic space. To address this problem, LDA-VAE (Srivastava and Sutton, 2017) instead used the logistic normal distribution to approximate the Dirichlet distribution. ProdLDA (Srivastava and Sutton, 2017) extended LDA-VAE by replacing the mixture model of LDA with a product of experts. SCHOLAR is a neural framework for topic models with metadata incorporation (Card et al., 2018). When metadata are document labels, the model infers topics that are relevant to those labels. Although some studies have applied the adversarial approach (Goodfellow et al., 2014) to topic models setting a Dirichlet prior on the generative network (Wang et al., 2019;Masada and Takasu, 2018), it is still unexplored how to use this mechanism to disentangle opinion-bearing topics from plot or neutral topics.
Compared to these neural topic models, DI-ATOM is the first attempt using an adversarial mechanism to distinguish between topic type (i.e. opinion and plot topics), while not only generating topics aligned with the available target labels (i.e. opinion topics) but seamless incorporating the external signal of plot summaries to drive the generation of topics about salient parts of plots mentioned by users (i.e. plot topics) not related to the target classes (i.e. sentiment polarity). Representation Disentanglement. Despite the lack of general consensus about a unique definition of disentangled representations (Higgins et al., 2018;Gao et al., 2019), it typically refers to representations which are only sensitive to one single generative factor of data and relatively invariant to other factors of variation (Bengio et al., 2013). One proposed definition builds upon the concept of statistical independence by minimizing total correlation (Dinh et al., 2015;Achille and Soatto, 2018), while an alternative approach explored the possibility to measure and track the changes in a single latent dimension as degree of disentanglement (Higgins et al., 2017). However, the disentanglement of representation achieved in DIATOM is instead analogous to the one presented in  and , where they impose additional constraints to the representations in the latent space that can be controlled exploiting a reinforcement learning mechanism determining the disentangled factors. In DIATOM, we alternatively make use of an adversarial approach over the available target labels.
Both Generative Adversarial Networks (GAN) (Mathieu et al., 2016;Makhzani et al.;Chen et al., 2016; and VAEs (Peng et al., 2019;Chen et al., 2018;Hsieh et al., 2018) have been successfully employed in disentangling features in computer vision tasks. Application in text processing has shown promising results (John et al., 2019;Kumar et al., 2017;Hoang et al., 2019;Esmaeili et al., 2019), yet applications to topic modeling are still limited (Wilson et al., 2016) and to the best of our knowledge, there is no work in separating opinion-bearing topics from plot/neutral topics.

DIATOM architecture
Assuming a document x is associated with a sentiment label y s , and each document can be represented by latent topics associated with sentiments (z s ) and plots 2 (z a ), we aim to learn a model maximizing the joint data-label log-likelihood, log p(x, y s ): Inspired by Miao et al. (2016) and Card et al. (2018), we assume the document-level topic distribution for plots can be approximated by a multilayer perceptron (MLP) taking as input a multivariate Gaussian distribution, and similarly for the topic distribution for sentiments. The multinomial distribution over words under a plot topic and an opinion topic can be parametrised by a weight matrix W . The generative process is shown below.
• For each document d ∈ {1, .., D}, where fφ, fψ and f y are MLPs, z a is a Kdimensional latent topic representation of plots for document d, z s is a S-dimensional latent topic representation of sentiments for document d. The probability of word x d,n can be parametrised by another network: where m d is the V -dimensional background logfrequency word distribution, and W ∈ R V ×(K+S) , while z a z s is the concatenation of the two latent topic vectors.
Plot Inference Network. Following the idea of VAE which computes a variational approximation to an intractable posterior using MLPs, we define two inference networks f µ φ and f Σ φ which takes as input the word counts in documents: The outputs of both networks are vectors in R K . Here, 'diag' converts a column vector to a diagonal matrix. For a document x, q(φ) LN (µ φ , Σ φ ). With such a formulation, we can generate samples from q(φ) by first sampling ∼ N (0, I) and then computingφ = σ(µ φ + Σ φ 1/2 ).
Sentiment Inference Network. Similarly, to compute a variational approximation to q(ψ), we define two inference networks f µ ψ and f Σ ψ which takes as input the word counts in documents: The outputs of both networks are vectors in R S . For a document x, q(ψ) LN (µ ψ , Σ ψ ). We then generate samples from q(ψ) by first sampling ∼ N (0, I) and then computinĝ ψ = σ(µ ψ + Σ ψ 1/2 ).
Overall Objective. With the sampledφ andψ, for each document x, we can minimise the reconstruction loss with a Monte Carlo approximation using L independent samples: where the first term in the RHS is given by Eq.
(2). It has been previously shown in (Kingma and Welling, 2014), if a standard multivariate normal prior is placed on the latent variables z a and z s , then there is a closed form solution to the KL divergence terms above. We assume that the latent topics associated with plots, z a , are independent of sentiment classes, and hence, when fed into a sentiment classifier, should generate a uniform sentiment class distribution (similar to adversarial learning). On the contrary, the latent topics associated with sentiments, z s , should bear essential information to discriminate between sentiment classes. Therefore, we define the following two objectives for sentiment classification; the former being the expected KL divergence with the uniform distribution U, and the latter a cross-entropy loss: where M is the total number of sentiment classes, and U(0, M ) represents the uniform sentiment class distribution.
To further disentangle the latent topics associated with plots and sentiments, while concurrently minimise the redundancy in the final topic matrix, we apply an orthogonal regularizer over the decoder matrix W . L orth reaches its minimum value when the dot product between different topic distributions goes close to zero: Our final objective function is: where α, β, γ and η control the relative contribution of various loss functions. Plot Network. An additional VAE is plugged to the model providing a supplementary signal for the latent plot topic extraction. This mechanism preserves the plot information that might contain some sentiment words and thus, be wrongly regard as a user's opinion. The inference network is defined analogously to Eq. 3, which instead of taking a review document, takes a plot summary as input. An additional cross-entropy objective is minimized to drive the latent plot topics (z a ) which would have a similar discriminative power as the features (z d ) directly derived from the plots when used for plot classification: where R denotes the total number of plots in each dataset. Finally, −L d and L plot are added to the overall loss defined in Eq. 8.

Experimental Setup
We conduct thorough experimental evaluations to assess the quality and disentanglement rate of extracted topics. To assess the quality of topics, we compute their topic coherence (Röder et al., 2015) coupled with their topic uniqueness. Then, we additionally look at the discriminative power of the disentangled features on the sentiment classification task. To fully assess the disentanglement rate of different methods, we perform topic labeling to compute the sentiment polarity of each topic (if any) and then measure the overall disentanglement rate (Eq. 11). As a result, we obtain an estimate of the extent to which different models can accurately control the topic disentanglement rate. We introduce and use a new dataset, named the MOBO dataset, pairing movie/book plots with their users' reviews, and including human-annotated sentences. MOBO Dataset. The MOBO dataset is a collection of reviews and plots about MOvie and BOok, associated to human-annotated sentences: while the pairs of reviews and plots are used to enhance the generation of plot topics, the human-annotated sentences provide the necessary ground-truth to automatically evaluate the topics' polarity. Movie and book reviews were collected and paired from 3 public datasets: the Stanford's IMDB movie reviews (Maas et al., 2011), the GoodReads (Wan et al., 2019) and the Amazon reviews dataset (McAuley et al., 2015). Among all the available reviews in the IMDB dataset, we keep the ones with a corresponding plot in the MPST dataset (Kar No. of annotated sentences 6,000 6,000 6,000 Goodreads dataset comes already with books' reviews paired with the related plots; while from the Amazon dataset, among all the product reviews, we keep only the ones related to movies available on the store and whose descriptions consist of the movie plots 3 . With the help of 15 annotators we further labeled more than 18,000 reviews' sentences ( ∼ 6000 per corpus), marking the sentence polarity (Positive, Negative), or whether a sentence describes its corresponding movie/book Plot, or none of the above (None) 4 . We ensured that each sentence was labelled by at least 2 annotators by assigning overlapping subsets of ∼ 2400 sentences. In case of disagreement, when no consensus was reached, a final choice was made through a majority vote involving a third annotator. The final interannotator agreement (Cohen's kappa) was computed between each pair of annotators sharing a common subsets, with a minimum value of 0.572 and maximum of 0.831, for a resulting average of 0.758 5 . We report the dataset statistics in Table 1. Baselines. We compare the experimental results with the following baselines: sLDA (Mcauliffe and Blei, 2008): a supervised extension of LDA adding a response variable associated with each document. JST (Lin and He, 2009): Joint Sentiment-Topic model built on LDA which is able to extract polarity-bearing topics. NVDM (Miao et al., 2016): Neural Variational Document Model, a variational auto-encoder with an encoder network (i.e. an MLP) mapping the bag-of-words representations into continuous latent distributions, and a generative network (i.e. a softmax decoder) reconstructing the document representations. GSM (Miao et al., 2017): based upon NVDM, the Gaussian Softmax topic model generating the topic distribution by applying a softmax function on the hidden representations of documents. NTM (Ding et al., 2018): Neural Topic Model is a variation of NVDM by plugging the topic coherence metric directly into the model's objective. PRODLDA (Srivastava and Sutton, 2017): with an architecture similar to NVDM, ProdLDA introduces a Dirichlet prior in place of Gaussian prior over the latent topic variable. SCHOLAR (Card et al., 2018): a neural framework based on variational inference for the generation of topic while incorporating metadata information. Parameter Setting. We perform tokenization and sentence splitting with SpaCy 6 . When available, we keep the default preprocessing, as it is the case for sLDA and SCHOLAR. Along with stopwords, we also remove tokens shorter than three characters and those with just digits or punctuation. We set the vocabulary to the 2,000 most common words as the best trade-off for each dataset.
The 300-dimensional word vectors are initialized with a pretrained BERT embedding (Devlin et al., 2019). Sentence embeddings are generated from the Sentence-BERT using a pretrained BERT-large with mean-tokens pooling (Reimers and Gurevych, 2019). We use the predefined split of the MOBO dataset into training, development and test set in the proportion of 80/10/10 and average all the results over 5 executions. Hyperparameters. We tune the models' hyperparameters on the development set via a random search over combinations of learning rate λ ∈ [0.001, 0.5], dropout δ ∈ [0.0, 0.6] and topic vector   25,50,100,200]. Encoder and decoder are configured following (Srivastava and Sutton, 2017). The hidden representation of documents is set to 100 and sentiment classifier's hidden size to 50. Matrices are randomly initialized with the Xavier and sparse methods (Glorot and Bengio, 2010;Martens, 2010). We employ the Adam optimizer (Kingma and Ba, 2015), set the batch size to 64 and apply batch normalization as additional regularizer (Cooijmans et al., 2017).
Sequential Unfreezing. Instead of simultaneously training all the model components, we unfreeze them sequentially. We first freeze the sentiment classifier and update only the autoencoder. At the e th epoch, we unfreeze the sentiment classifier uniquely on the polarized features to let the classifier training. Finally, at the (e + n) th epoch, we unfreeze the adversarial mechanism to drive the generation of neutral features fooling the classifier. We follow an analogous approach with regards to the plot classifier. The values of e and a are treated as hyperparameters and chosen through the random search. We found that the sequential unfreezing scheme leads to better topic disentanglement.

Experimental Results
We report the results in terms of topic coherence/uniqueness, sentiment classification and topic disentanglement rate. We also perform ablation studies to gain more insights into our model.

Topic Coherence and Uniqueness
Traditionally, topic models have been evaluated through the perplexity over held-out documents (Wallach et al., 2009). Lower perplexity implied better predictiveness as it aimed at measuring the model goodness-of-fit over a held-out set. However, it has been shown that better perplexity does not imply more comprehensible topics (Chang et al., 2009;Card et al., 2018) . That is why topic coherence was introduced (Lau et al., 2014), to evaluate topics regarding their understandability with a score closely matching human judgments. Normalized Pointwise Mutual Information (NPMI) is a topic coherence score shown effective in matching the human judgments (Röder et al., 2015), and measures the statistical independence of observing two words in close proximity based on the word co-occurrence statistics. The NPMI for a list of words w is defined in Eq. 10: where P (w i ) and P (w i , w j ) are calculated based on the word co-occurrences in a reference dataset, and N is typically set to 10, thus considering the top-10 words of topics. The aforementioned definition normalizes the PMI in [−1, 1], so that for two words, -1 denotes no co-occurrences while +1 a complete co-occurrence. We evaluate topic coherence using the C V metric, a slightly refined NPMI score using a boolean sliding window to determine the words' context (Röder et al., 2015).
We additionally monitor the topic uniqueness (TU) to measure word redundancy across topics. Following Nan et al. (2019), we use cnt(l, k) to denote the total number of times the top word l in topic k appears among the top words across all topics, then T U (k) = 1 L L l=1 1 cnt(l,k) . TU is inversely proportional to the number of times each word appears in topics; a higher TU score implies that the top words are rarely repeated and, therefore, more diverse and unique topics.
In Table 2, we report the topic coherence and the topic uniqueness values. The supervised document label information was incorporated into sLDA, JST, SCHOLAR and DIATOM. Other models are purely unsupervised. We can observe that among conventional LDA-based models, JST performs significantly better compared to both LDA and sLDA for different topic settings and across all datasets. Neural topic models give mixed results. In terms of topic coherence, the trend is SCHOLAR > PRODLDA > NTM > GSM > NDVM. However, when we examine the topic uniqueness values, we can see that higher topic coherence values do not necessarily lead to higher topic unique-ness values. This shows that the topic coherence value could be misleading sometimes since a high topic coherence could be due to the redundancy of words across topics. We also notice that models with supervised document label information (except sLDA) generally outperform the unsupervised ones. This shows that the document label information can indeed help to extract more meaningful topics. When compared our proposed DI-ATOM with the baselines, we can observe that it achieves better coherence and topic uniqueness values most of the time, showing the benefit of separating opinion-bearing topics from plot topics by adversarial learning. The importance of the plot network is evident from the results since removing the plot network ("−w/o Plot Network") leads to the degraded topic coherence and topic uniqueness measures.

Sentiment Classification
In this section, we compare DIATOM with other supervised topic models for sentiment classification. The purpose of this evaluation is to highlight the discriminative power of the generated representations for the labels of interest while having attractive and unique properties as topic models, rather than confronting them with current state-of-the-art for text classification. We additionally report some baseline results using a Support Vector Machine (SVM) which has been widely employed on these task (Pang and Lee, 2004b) providing an understanding of the relative differences in performance of different approaches. Table 3 shows the sentiment classification accuracy. In JST, the supervised document label information is only incorporated as prior to the model, while both sLDA and SCHOLAR treat the class label of each document as a response variable and jointly model both documents and their responses. We can observe that the latter is more effective in incorporating supervised information since both sLDA and SCHOLAR outperform JST in general. But DIATOM gives significantly better results all over the baselines with the improvement over the best baseline model, SCHOLAR, by 3-8%. In our models, features used for sentiment classification are opinion-bearing topics. This shows that separating opinion topics from plot/neutral topics is beneficial for sentiment classification. We also observe that the contribution of the plot network to sentiment classification is dataset-dependent. The use of plot network largely boosts the sentiment  classification accuracy by over 8% on the Amazon dataset. But its effect is negligible on the other two datasets.
When compared with traditional sentiment classification models such as SVM, we found that DI-ATOM outperforms SVM trained with various features on both IMDB and Amazon. But it performs slightly worse than SVM trained with TFIDF features with or without an additional incorporation of sentiment lexicon features. Nevertheless, DI-ATOM gives superior performance compared to SVM trained on LDA topic features in the range of 5-11%, showing the effectiveness of using opinion topics for sentiment classification.

Topic Disentanglement
None of the aforementioned measures can, however, capture how opinion and plot topics are distributed. To this aim, we use topic labeling to assign a proxy label (Positive, Negative, Plot, None) to each topic and then measure the topic-disentanglement rate ρ (Eq. 11) as the proportion of opinion-bearing topics with respect to the overall set of topics, complementary to the proportion of plot/neutral topics: with S being the number of opinion topics and K the number of plot/neutral topics. For each topic, we first calculate its embedding by taking the normalized weighted average of the vectors of its top N words: where α i is the normalized distribution of word w i in topic z. We then retrieve the top 10 most similar sentences from the human-annotated sentence set measured by the cosine similarity between the topic embedding and each sentence embedding.
The sentence embedding is computed using the Sentence-BERT encoder (Reimers and Gurevych, 2019). The most frequent label among the retrieved sentences is adopted as the topic's label.
To highlight the disentanglement capability of DIATOM, in Figure 3, we analyse how the proportion of opinion-bearing topics varies across standard and sentiment topic models. We notice that despite the signal from the document labels, sLDA and SCHOLAR tend to produce topics rather balanced in terms of neutral and opinion-bearing topics. JST has a more skewed distribution towards opinion topics. DIATOM instead generates an actual proportion of opinion topics approaching the expected proportion set up by the model, demonstrating the capability to control the generation of plot and opinion-bearing topics.
In Table 4, we show a set of topics grouped according to the disentanglement induced by DI-ATOM. For each topic, we report an excerpt of the most similar sentences retrieved. Aside from being overall coherent, we can guess rather paradigmatic themes as the IMBD-Topic 1 about peace and war between countries, or more peculiar plots related for instance to "The Hobbit" or "Batman" as in the Amazon topics. It is worth having a closer look at the IMDB-Topic 2, which despite the "negative" theme of depression and suicide, the model is able to correctly gather those words under the same plot topic. The opinion-bearing topics report a collection of commonly appreciated or critic aspects; some of them are mainly collections of related adjectives with the same polarity (e.g. IMDB-Topic 1), while others are made up of mixed terms describing the issues and the associated experience (e.g. Amazon-Topic 2).

Visualization
Another way to look at the disentangled topics is through the visualization of topic vectors.
As an example, we plot in Figure 4 the 2dimensional representation of the topic distributions projected by t-SNE for the Amazon dataset. Different colors represent different types of topics generated by DIATOM, namely plot/neutral in blue and opinion in red. We notice how consistently across different number of topics, plot/neutral topics tend to cluster together, with the boundary close to some polarized topics likely to share common features, as shown in Figure 1 in which the plot topic and the negative topic share a common word 'Dwarf '.

Further Discussion
Ablation Study. We report in Table 5 the results of the ablation study on DIATOM. We observe that removing the orthogonal regularization has a limited effect on sentiment classification, but causes a fluctuation on topic coherence and a clear drop in topic uniqueness. A significant classification performance drop is observed by removing the sentiment classifier, which essentially reduces DIATOM to an unsupervised model. Removing both the orthogonal regularization and the sentiment classifier shows a major negative impact on both accuracy and the topic's quality.
Finally, we assess the influence of the plot network ( §3), and while we do not notice any consistent impact across the datasets in terms of sentiment classification, the quality of topics has a notable drop in terms of coherence and diversity.  Table 5: Ablation study over DIATOM by removing the orthogonal regularization, the sentiment classifier or just the auxiliary Plot Network.
Limitation and extension. Although the adversarial mechanism implemented in DIATOM is rather effective in disentangling opinion and neutral/plot topics, at times the opinion topics could exhibit terms of mixed polarities. An additional adversarial mechanism can be a viable solution at the cost of increasing the model's complexity.
Moreover, in our current model, the latent plot topics z a extracted from reviews are encouraged to have a similar discriminative power as the latent topic z d learned from plots directly for predicting the plots. It is also possible to impose a Gaussian prior centred on z d for the latent plot topics in reviews instead of using the Gaussian prior of zero mean and unit variance. Another approach is to replace the plot classifier with a discriminator as typically used in GAN training that the learned plot topics from different sources (reviews and plots) are competing with each other to confuse the discriminator.
While we focus on separating opinion topics from plot or neutral ones in movie and book reviews in this work, our proposed framework can be applicable in other scenarios. For example, in veracity classification of Twitter rumours, we want to disentangle latent factors which are indicative of veracity of tweets from those which are event-related. Our proposed framework provides a potential solution to it.

Conclusions
We have described DIATOM, a new neural topic model to generate disentangled topics through the combination of VAE and adversarial learning.
We reported the results of our experimental study based on the novel MOBO dataset highlighting the benefit of such an approach leading to topics with higher interpretability in terms of both topic coherence and topic uniqueness and more discriminative power reflected in better sentiment classification results compared to other supervised topic models.
We further discussed the model capability to consistently disentangle opinion-bearing topics from plot/neutral ones measuring the introduced disentangling rate. Finally, we identified current limitations and viable solutions to be explored in the future.