Studying the Evolution of Scientific Topics and their Relationships

We propose a study of the development of scientiﬁc topics through time, as well as the relations between them within the scientiﬁc ﬁeld of computational linguistics and across subﬁelds. We use topic modeling to analyze scientiﬁc texts published in the ACL Anthol-ogy, and introduce a categorization of topics in our ﬁeld into 3 types: tasks, algorithms, and data. In order to understand how topics emerge, evolve, and gradually disappear over time, we analyze the evolution of these topics across time through several case studies. We further include in our analysis papers published in NeurIPS, and try to understand whether there was any inﬂuence between topics in this conference focused on neural meth-ods and computational linguistics conferences, as well as measure the divergence over time between conferences in terms of the topics approached. We additionally look at the relationships between topics, categorizing them into types of competing or cooperating topics.


Introduction
Scientific fields progress through innovation. Science functions under the premise that, when new better topics appear in research, they overtake the old ones and contribute to shaping the progress of the research field (Kuhn, 2012). Nevertheless, scientific topics evolve interdependently (the appearance and popularity of one topic may affect the popularity of another) and oftentimes, the focus of research in a certain field is also influenced by topics in other related subfields.
We propose a multidimensional approach for studying scientific topics and their evolution, by analyzing our field of research -computational linguistics -from several points of view: we look at the parallel evolution of topics in computational linguistics and their popularity over time, as well as how they relate to each other, engaging in cooperating or competing relationships. We also extend this perspective by considering the interplay of topics within a field, as well as the context in which they appear, and how the same topic is portrayed in different subfields, with a focus on the mutual influence between ideas in computational linguistics and those in the related field of neural networks.
Among studies that track the evolution of topics in scientific texts, Hall et al. (2008) focused on scientific text in computational linguistics, analyzing papers published in ACL, EMNLP andCOLING between 1978 and2006. The authors identify increasing and decreasing trends up to 2006, and make predictions about the subsequent evolution of the field. We continue the analysis including articles published up until the end of 2018, and uncover current shifts and trends that may not have been predictable 15 years ago -such as the rise of neural networks methods.
In our work, we study topics across three types: tasks, algorithms and data. Moreover, our aim is to further and complement the previous explorations of topics in computational linguistics not only by extending the analysis to recent years, but also by looking at relations between topics within and across fields. We analyze texts in four top computational linguistics conferences (adding NAACL to the three conferences analyzed in (Hall et al., 2008)). We additionally propose an exploration of topics across conferences and subfields, and include in our analysis papers published in the Conference on Neural Information Processing Systems (NeurIPS), which is a machine learning conference focused on neural networks. Considering that in recent years neural networks have almost dominated methods used in computational linguistics, we try to understand how topics approached in computational linguistics relate to those in the more focused field of neural networks, and whether and how they migrate between these conferences.
Our analysis of topic relationships within computational linguistics is inspired from Tan et al. (2017), in which the authors propose a way to classify topic relationships into four types, based on their co-occurrence in text and the degree of correlation between their popularity over time. In our paper, we extend this and take a deeper look at the relations existing between topics in scientific text. We propose interpretations of topic relationships in the context of a scientific domain, and report interesting findings on how these types of relationships manifest between scientific topics, discovering, for example, which algorithms in computational linguistics are compatible with certain tasks (such as neural machine translation and RNNs), or finding pairs of topics that represent algorithms which have replaced one another along the history of computational linguistics (such as statistical machine translation and neural machine translation).

Previous work
Multiple previous studies have looked at evolution of topics through time, analyzing texts of various genres, from news (Michel et al., 2011;Rule et al., 2015) to emails (Wang and McCallum, 2006) to scientific articles (Hall et al., 2008;Prabhakaran et al., 2016;Griffiths and Steyvers, 2004;Blei and Lafferty, 2006;Anderson et al., 2012).
Popular choices for representing topics include topic models, to which some studies add variations specific to tracking trends over time, such as the continuous-time model proposed by Wang and McCallum (2006), the generative model proposed by Bolelli et al. (2009a,b), or the dynamic topic model (Blei and Lafferty, 2006). Hall et al. (2008) use an approach based on topic modeling, and focus on scientific texts in computational linguistics, analyzing papers published in ACL, EMNLP and COLING between 1978and 2006. Gollapalli and Li (2015 use topic models and keyphrase extraction to compare topics in ACL and EMNLP. In other studies on scientific articles, topic representations are enriched with additional features such as citations (He et al., 2009). Citations and citation networks have been leveraged extensively in previous studies for tracking scientific topics (Shibata et al., 2008(Shibata et al., , 2009Jurgens et al., 2018), analyzing the structure of the scientific community (Leicht et al., 2007), or summarizing scientific papers (Qazvinian and Radev, 2008), or entire tech-nical topics . Other authors make use of rhetorical framing to predict the patterns present in the development of scientific topics (Prabhakaran et al., 2016).
Not as many studies attempt to provide in-depth systematic analyses of the relations between topics within a field or across fields, independently from the publications where they occur. Zhang et al. (2017) introduce a learning technique to identify the evolutionary relationships (e.g., topic evolution, fusion, death, and novelty) between scientific topics. Grudin (2009) study the particular relationship between the field of AI and Human Computer Interaction. Shi et al. (2010) propose a temporal comparison of grant proposals and academic publications, in an attempt to understand which precedes the other and how they influence each other. In one of the most extensive studies on the topic (Tan et al., 2017), the authors propose a systematic way of classifying relations between topics into four types of cooperating or competing topics, based on their patterns of co-occurrence and prevalence correlation: friendships, arms-race, head-to-head, and tryst. We build on this framework in our analysis of the field in the following sections.

Dataset
Our study focuses on topics in computational linguistics and their evolution. For exploring this topic, we make use of articles published in the ACL Anthology (Bird et al., 2008; from its inception. We collect all papers published in four top conferences: ACL, EMNLP, COLING and NAACL over time, obtaining a total of 14,737 computational linguistics articles overall. We will further refer to the set of computational linguistics conferences we considered by using the general term ACL+. For the second stage of our study, we additionally use articles published in the NeurIPS conference, from which we collect all articles published since 1994, in total 6,520 articles. Table 1 shows the number of articles for each time period (across 5-year time spans) for the ACL+ conferences and NeurIPS. In Figure 1 we show the number of papers published as a time series, computed separately for each of the conferences considered. We make our collected dataset as well as code used for our experiments publicly available. 1 pre-1980 374 -1980-1985 332 -1986-1990 729 -1991-1995 609  157  1996-2000  1108  842  2001-2005  950  767  2006-2010  3456  1449  2011-2015  3432  1091  2016-2018 3747 2214

Representation of ideas
We base our study on the premise proposed by Kuhn (2012) that science proceeds by shifting from one paradigm to another, viewing the evolution of science as a series of topics that follow and replace one another. Furthermore, we assume that these shifts in topics are directly reflected in shifts at the level of the vocabulary employed in the articles that discuss them. Based on this assumption, we choose to represent topics by relying on topics extracted using unsupervised topic modeling, which treats documents as bags of words generated by one or more topics. We choose to measure the topics' evolution over time post-hoc, using a classical topic model and monitoring the change in topic prevalence over time. While dynamic topic models (Blei and Lafferty, 2006) allow to include temporal information in the generated labels themselves, they impose additional constraints on the time periods (for example assuming the changes between consecutive years are the same). We design our representation of topics starting from the observation that computational linguistics research can generally be described as comprising of a set of research tasks, which researchers aim to solve by employing appropriate algorithms, usually assisted by the use of datasets. Based on this assumption, we propose that topics in computational linguistics can naturally be categorized into 3 types: tasks, algorithms and data. As such, we propose a notion of scien-tific topic in our field which consists of both a topic and its category or type; in this view, a topic in computational linguistics can be defined as: typec ∈ {task, algorithm, data}, topict ∈ T, with T representing the list of topics generated by the Latent Dirichlet Allocation model (LDA) (Blei et al., 2003). These topic categories can be useful beyond our field and application, for example in question answering systems or paper recommendation systems (Augenstein et al., 2017;Park and Caragea, 2020;Luan et al., 2018;QasemiZadeh and Schumann, 2016). In order to identify the topics occurring in our corpus of scientific texts, we first train an LDA model on the full texts extracted from computational linguistics articles, and use it to extract a set of 100 topics which we will use to analyze the evolution of the field in the next stages of our study. We use the Mallet implementation of LDA 2 , with parameters set to 100 topics, and 100 training passes. The asymmetric prior distribution was learned directly from the data. The resulted model has a topic coherence score of 0.484 according to the C V coherence measure. In order to maximize the quality of the produced topics, we first label the obtained sequences with POS tags and select only words with POS tags corresponding to content words: nouns, verbs, adjectives, and adverbs, and discard the rest. We lowercase and lemmatize the texts, and we extract bigrams and trigrams using PMI scores to select words which occur together with high probability and add them to our vocabulary and document representations. On the collection of articles published in the ACL Anthology preprocessed as described above, we train the topic model to extract 100 topics. We do not intervene with significant changes on the output of the model, and only add minor corrections, through manual curation: we remove 10 of the extracted topics which we do not consider to represent coherent or interesting ideas, and merge a few topics which were redundant. We are left with a total of 82 topics.
We then manually label each topic with one of the three proposed categories: task / algorithm / data, and obtain the final list of topics occurring in our corpus. Each topic can be assigned one or more types: we obtain 53 topics labelled as tasks, 33 of the topics are algorithms, while 7 topics fall into the data category. Some topics belonging to the task type are, for example, morphology, event extraction, or summarization. Topics such as recurrent neural networks or topic models fall under the category of algorithms, whereas lexicons and parallel corpora are categorized as data (or resources). A few topics refer to inherently connected tasks and algorithms, we label those with both types -as is the case of neural machine translation or statistical machine translation. The appendix lists the entire set of extracted topics, along with the top 10 keywords that are relevant for each, as well as their types. When topics were merged, the list of keywords relevant for each topic were merged into one larger list.
After having generated our list of topics, we further extract for each paper a list of relevant topics, considering only those which are present in the topic distribution for that document with a probability greater than 0.01. After this step, we are left with almost 13 relevant topics per article, on average. Finally, we measure the prevalence of a topic during a certain year by computing the empirical probability of its occurrence relative to the total number of topics that were approached overall in that year: where I is the indicator function, t d is the year in which document d was published. The conditional probability of a topic given a document P (t|d) is thus equal to 1 if the topic is present in the document and 0 otherwise. C y represents the total number of documents written in a year y. Figure 2 illustrates the distribution of topics across the computational linguistics corpus for each of the 3 topic types. Although the list of topics is

Selected topics and trends
In order to narrow our focus to subsets of topics worthy of interesting insights, we propose a few ways to select topics that stand out and comment on their development over time -several case studies will be presented in the following subsections. We also look into the most influential authors for each topic. We consider citations as an indicator of the influence of an author over a topic, and we thus measure the influence of each author for a topic by counting all of the occurrences of citations referring to the given author (regardless of the topic of the cited article) in all documents in our collection where the topic is present. Table 2 shows the top 5 most influential authors, ranked by number of citations, for a selection of topics.
Confirming and refuting predictions We first confront our findings with the predictions made in previous studies which looked at the evolution of scientific ideas in computational linguistics. Hall et al. (2008) identify a list of topics which were then on an increasing trend in 2006: classification, probabilistic models, statistical parsing, statistical machine translation and lexical semantics. We find among our topics those which best match their list, then analyze their evolution in order to see whether the predictions made then still hold today. Figure 3 shows the evolution of four of these topics  until 2018: not all of the topics have maintained the same upward trend all through 2018. Statistical machine translation and probabilistic models suffer a decrease in popularity after 2010; classification, though still very popular, has reached a plateau, while lexical semantics seems to be still on an increasing trend, though less abruptly.
Most prevalent topics In our second case study we focus on the most prevalent topics overall, which we consider to be ones that over time have received the greatest attention in computational linguistics research. To find these, we average the probability of occurrence of a topic in each year, obtaining for each topic an overall score of prevalence: Prev(t) = 1 |Y | y∈Y P (t|y) Figure 4 shows the evolution of the top 5 most prevalent topics in ACL+ across time. Most of these were very popular in the earlier days of computational linguistics and started to decrease around 1990, such as the topics related to syntax. Complexity analysis has a steady evolution across time, maintaining a relatively flat trend.
Topics with largest variation In our next analysis, we extract topics which vary most in popularity over time, hoping to discover topics which stand out because of their dramatic evolution over time. We do this by considering the distribution of probabilities for a topic over the years, and measuring its standard deviation, for each topic, then select those topics where standard deviation is highest. The top 5 such topics and their evolution are illustrated in Figure 5. It seems that the most dramatic variations are related to recent increases in popularity of certain topics, most of which relate to machine learning. The steep and constant increase in popularity of the learning topic is apparent. Among the first 5 topics which vary most dramatically in popularity over time we find topics related to neural networks, which although very recent relative to the entire history of computational linguistics, have quickly caught up in popularity and even surpassed more traditional topics in the field, and show an abrupt increase in popularity after 2010. We analyze topics related to neural networks in more detail in the following paragraphs.
Neural networks In our final case study, we zoom in specifically on topics related to neural methods. These are shown in our previous results to be the stars of recent years in computational linguistics, showing an abrupt increase in popularity.
The list of topics generated by our LDA model produce no less than four distinct topics related directly to neural networks, found in computational linguistics papers, which is already remarkable for such a recent topic. These are: neural networks, recurrent neural networks, neural machine translation and embeddings. To these we add for our analysis the topic of learning, as the general class of topics under which neural networks fall, and whose evolution we also expect to be affected by the popularity of neural networks.
Furthermore, we compare the trends of neural network related topics in ACL+ to the same trends present in a conference focused primarily on neural networks: NeurIPS. In order to achieve this, we use our LDA model trained on ACL+ papers to extract topics from NeurIPS papers. Figure 6 shows the evolution of topics related to neural methods in papers published in ACL+ and in NeurIPS, respectively. Both papers in ACL+ and in NeurIPS show the same steep increase in recurrent neural networks and neural machine translation starting between 2010 and 2015. Learning has a clearly more stable evolution in NeurIPS, where it has been a very popular topic from the beginning, as compared to computational linguistics, where it sees a steady and still continuing increase. Interestingly, neural networks as a general topic evolve differently in NeurIPS and ACL+: while in computational linguistics they are a recent topic, with a sudden increase in popularity after 2010, in NeurIPS they were widely discussed from 1994, and have suffered a decline up to 2010 when they started following the same upward trend.

Relationships between Topics
Methodology We use measures of relatedness between topics on two dimensions: co-occurrence and prevalence correlation, to characterize relationships into four major types of relations, which will be described and interpreted in more detail in this section: friendship, head-to-head, arms-race and tryst.
For categorizing pairs of topics into these types of relationships, we obtain co-occurrence scores for a pair of topics by computing the PMI score for the topics as they co-occur in documents, and compute the correlation score as the Pearson correlation between the time series represented by the topic's probability over time.
We then split each of these two dimensions into two classes (positive/negative co-occurrence, and positive/negative correlation), obtaining the four types of relationships from their combinations. We first standardize the distributions of co-occurrence and correlation scores, then split the relations landscape into four parts, depending on where they are situated on the two axes: positive/negative cooccurrence and positive/negative correlation.
We also compute a measure of strength of each relationship between a pair of topics, which is simply the product of the two scores, in absolute value. Sorted by the average strength of top 25 relations of that type, the relations rank as follows: friendships>head-to-head>tryst>arms-race. Table 3 shows the top pairs of topics with the strongest relations for each relation type, as well as their strength. The appendix contains tables with the top 10 relations for each relation and topic type.
We separately identify relations between different types of topics, and propose that some relations are more meaningful for certain topic pairings than others, depending on their types. For friendships, which refer to cooperating topics, we focus on topic pairs of different types, between which this relation is established, in order to discover the tasks go together with specific algorithms or datasets. For the other relation types (arms-race, head-to-head and tryst), we suggest that the cross-type topic pairs are less meaningful -since these types of relations can be interpreted as occurring between competing topics -for these we focus instead on same-type topic pairs (between tasks and tasks, algorithms and algorithms, data and data). In the tables presenting top relationships for each type, we restrict our focus to only topic pairs of types which can be meaningfully matched for each relation.
Friendships Two topics are "friends" if they tend to co-occur in the same texts and are also correlated in their prevalence over time. These are topics which go together, or "cooperate" -they are often found in the same documents and are used together in the analysis of a certain idea or area of interest. Figure 7 shows the top strongest friendship relationships between a task and an algorithm, and an algorithm and data, respectively. We discover, for example, that the neural machine translation task is most associated with the recurrent neural networks algorithm, and that for the task of statistical machine translation, parallel corpora are the most   useful types of datasets.
Head to head Topics in a head-to-head relationship do not tend to co-occur in the same documents, and are anti-correlated over time. These are topics which have nothing in common, or are even rivals.
In Figure 8 we can see the strongest head-to-head relationships in our corpus between tasks and algorithms respectively. One example is the relation between grammars and neural machine translation: these are rarely treated together in studies; more than that, while neural machine translation shows a recent increase in popularity, grammars are on a declining trend.
Arms race An arms-race relation characterizes topics that are correlated in their usage over time, but do not tend to co-occur within the same documents. Topics in this type of relationship tend to evolve in a similar pattern over time, possibly with an underlying common cause, even though they are not directly related: such is the case of many algorithms which were widely used before being recently replaced by neural networks. Figure  9 shows two such pairs of topics: phonology with semantic role labelling, and topic models with dependency parsing, which show similar decreasing trends, but are not referred to in the same articles.
Trysts Tryst is a relationship between topics which tend to co-occur in the same texts, but are anticorrelated in prevalence over time. We show that according to our study, this is one of the most interesting relations occurring between scientific topics, Figure 9: Examples of topics in arms race relationships and their evolution over time. and propose that it is useful for discovering topics that are replaced by others: topics which share a common niche of the research field, but as one topic increases, the other decreases. In Figure 10 we see two such relationships, which uncover interesting topic pairs. One is statistical machine translation versus neural machine translation, which is clearly a topic in the subfield of machine translation which has recently replaced the previous one as the primary focus of researchers. A similar phenomenon may have occurred for data-typed topics related to language resources: while dictionaries are overall more studied, they are on a decreasing trend, and have now been surpassed in popularity by parallel corpora.

Relations between conferences
Conference divergence In this part of our study we focus on the relations between conferences in computational linguistics. We compute divergence between conferences using Jensen-Shannon divergence applied on their topic distributions generated by papers published in each conference. Jensen-Shannon divergence is computed as the average of (a) ACL+ conferences.
(b) ACL+ vs NeurIPS. the KL divergences between each of the distributions and the average of the distributions. Its value is 0 for identical distributions, and tends to infinity as the two differ more and more. Figure 11 shows the pairwise divergence over time between the computational linguistics conferences, as well as between the linguistics conferences and NeurIPS. The span of each pairwise divergence plot is limited to the span of the youngest conference in the pair; the values are smoothed using a rolling average with a window of 2 years. The plot reveals a decreasing trend for all conference pairs. ACL and COLING are the conferences with the oldest history, and show a steady but mild decrease in divergence throughout their evolution. The most similar conferences are shown to be ACL with EMNLP and with NAACL, which also show the steepest decrease in divergence.
We further extend our study to contrast the computational linguistics conferences with NeurIPS. It is interesting to see that, even though computational linguistics and neural methods are technically distinct fields, the linguistics conferences still tend to converge with NeurIPS over time (although the absolute divergence between these is still considerably higher than among computational linguistics conferences). The most similar conference to NeurIPS in terms of the topics approached seems to be EMNLP, which from its beginning was the closest to NeurIPS among all linguistics conferences. This is perhaps explained by the more applied character of EMNLP compared to the others. In contrast, COLING, the oldest and most linguistics-focused of the conferences, is the least similar to NeurIPS, although still shows a tendency towards decreasing this gap.

Synchronicity of topics across conferences
Next, we introduce a second measure of similarity between conferences, this time over particular topics, in order to understand if conferences are synchronized in the topics they approach, and if this depends on particular sets of topics. Similarly to  the measure of correlation used in the topic relationship analysis, the correlation between conferences for a subset of topics T is simply computed as the prevalence correlation of topics over time, on average, for each topic in the subset considered -this time between its evolution in the two conferences (or sets of conferences) to be analyzed.
where the correlation between two conferences for a certain topic t is defined as: y (P (t|y, c 1 ) − P (t|y, c 1 )((P (t|y, c 2 ) − P (t|y, c 2 )) y (P (t|y, c 1 ) − P (t|y, c 1 ) 2 y (P (t|y, c 2 ) − P (t|y, c 2 ) 2 Using this measure we try to analyze how similar topics appear in different conferences over time, whether they follow similar trends or even influence each other.
With an average correlation across all topics between NeurIPS and ACL+ of 0.71, this measure also shows a fairly similar evolution of topics between the conferences overall. We should note however that the topics used in the analysis were generated only from ACL+ papers, so topics exclusive to NeurIPS are not considered. We then rank the topics in our list by the correlation of their evolution in NeurIPS versus ACL+: 5 topics among the top 10 with the most correlated evolution are shown in Table 4.
Neural topics in computational linguistics versus NeurIPS Neural networks are an interesting subset of topics, which have very quickly become very popular in computational linguistics, and are today common as central foci of both ACL+ and NeurIPS. The average correlation between ACL Anthology and NeurIPS for topics related to neural methods (neural networks, RNNs, neural MT and embeddings), is 0.58, which interestingly is lower than the overall correlation across all topics.
We try to understand whether these conferences are synchronized in the way they approach topics and hope to understand, by comparing their evolution, if they mutually influence each other, especially regarding topics which are relevant for both. In order to analyze this phenomenon, we compute the correlation between the evolution of topics, this time introducing an artificial lag for the papers in ACL Anthology. The correlation of topic time series is computed using an updated definition of topic probability: where l is a lag factor. Figure 12 shows the correlation between the evolution of topics after applying lags ranging from −25 to 25 years, for the full set of topics, as well as for the subset of topics related to neural networks. If there is any asynchronicity in the way topics appear in the two fields, the lag corresponding to the best correlation should help us find the delay with which topics gain popularity in the two conferences comparatively. In our case, the optimal lag value across all topics is found to be exactly 0, whereas for neural topics the optimal lag is 1 year, showing a slight delay in the approach of neural method related topics in ACL+. Overall, ACL Anthology and NeurIPS seem fairly synchronized when in comes to innovation in this area.

Conclusions
We presented in this article an analysis of the topics found in computational linguistics conferences. We enhanced topics with their types by categorizing topics into tasks, algorithms, and data; and showed how the field has evolved, uncovering general trends, as well as new unforeseen trends such as the abrupt rise of neural network methods. We also identified the most influential authors for each topic, which can provide interesting insights assuming most cited authors when discussing an idea carry a big share of the responsibility of introducing and promoting the idea. A more sophisticated method for identifying influential authors could include a normalization factor based on the number of citations.
We additionally included a study of relations between topics and between subfields, to gain insight into the interplay between topics within and across fields. Our analysis confirmed the strong cooperative relationship between certain tasks and algorithms, such as neural machine translation and recurrent neural networks, but also revealed some interesting less obvious ways in which some topics relate -automatically identifying topics which replace others in the preference of scientists in a subfield (such as the change in paradigm for machine translation). In a separate experiment, we zoom in on the topic of neural networks, and compare the evolution of this topic in computational linguistics conferences to its parallel development in a conference dedicated to neural networks: NeurIPS. Through the various complementary analyses we performed, we try to contribute to answering the question of how scientific topics emerge and gain traction by considering internal as well as external factors, and the scientific context in which trends appear and evolve. In the future, we would like to explore predictive models of what research topics would gain popularity in upcoming years. It would also be interesting to explore the effect of extracting more fine-grained topics, which could help with identifying more subtle trends -at the technical level, this would involve controlling the level of noise when increasing the number of topics. We will also explore in more depth the properties of the emerging network of topic relations, and the types of topics involved. Exploring more complex topic structures could help model more sophisticated notions such as scientific paradigms. A Full list of topics Domain adaptation (task) domain adaptation adapt cross data share weight distribution multi scenario Automata (algorithm) string transformation finite operation transducer stre match regular weight symbol Morphology (task) morphological arabic morpheme stem suffix morphology prefix root affix inflection Multi-word expressions (task) expression collocation literal metaphor idiom mwe multiword descriptor mwes compositional Sentiment analysis (task) sentiment negative positive opinion polarity lexicon subjective classification subjectivity neutral Trees (algorithm) tree node child root subtree parent forest leaf branch depth Reinforcement learning (algorithm) action agent dialog policy reward instruction environment goal human reinforcement SVMs (algorithm) kernel svm bag vector bow space reranke linear clue support Linear programming (algorithm) constraint solution variable solve inference constrain ilp hard linear soft Argument mining (task) claim essay argument stance email evidence support debate statement topic Topic models (algorithm) document topic lda collection distribution topical latent content coherence background Clustering (algorithm) cluster clustering group induce merge class partition gold induction centroid Language acquisition (task) student author learner simplification write native readability grade complex read Generation (task) generation generator content record surface realization choice plan selection component Named entity recognition (task) token joint ner span crf sequence labeling normalization pipeline crfs Discourse segmentation (task) segment segmentation boundary unit length break sequence segmenter segmented window Events/temporal (task) temporal anchor event tense expression interval causal date day reference Phonology (task) letter phoneme syllable pronunciation phonetic vowel phonological stress consonant sound Stylistics (task) emotion social gender age group emotional participant people relationship person Unification (task) grammar unification head formalism cat description hpsg sign definition constraint Language models (task) gram probability bigram lm perplexity trigram unigram estimate vocabulary smooth Textual entailment (task) entailment inference hypothesis game textual player rte premise entail team Biomedical (task) cue medical citation abstract patient scientific scope biomedical cite article Anaphoral/coref. resolution (task) pronoun mention antecedent coreference resolution coreference resolution anaphor resolve anaphoric reference Dependency parsing (algorithm) dependency parser parse head treebank tree dependent projective arc accuracy Database/resources (data) template database logical hybrid variable city expression meaning sql equation Social media/web data (data) user response post comment message conversation thread interaction feedback reply Summarization (task) summary summarization document rouge compression content length extractive human duc Spelling correction (task) error edit correction spelling revision rate confusion preposition incorrect learner Evaluation/annotation (task) human metric paraphrase reference correlation quality automatic judgment judge rating annotation annotator annotate agreement annotated gold scheme guideline automatic manual Semantic role labelling (task) argument predicate role arg srl syntactic identification propbank labeling core Discourse (task) discourse relation coherence connective implicit unit explicit paragraph marker rhetorical Syntactic structure (task) noun adjective compound head modifier modifi nominal determiner proper adverb verb subject object class preposition verbal noun passive argument syntactic syntactic linguistic syntax grammatical construction structural lexical deep surface phenomenon Lexical semantics (task) similarity vector cosine distributional sim distance weight relatedness space lsa Learning (algorithm) weight log parameter objective loss update optimization linear optimize paramet Probabilistic models/distributions (algorithm) distribution probability sample variable latent prior parameter estimate inference generative Statistical MT (task,algorithm) alignment align link probability ibm aligned null correspondence aligner heuristic translation translate quality mt target statistical translator smt reference bilingual translation bleu reorder decode smt hypothesis side decoder target chinese Transfer learning (algorithm) target transfer projection mapping project side map direct ds auxiliary Speech recognition (task) speech recognition speaker asr speak utterance acoustic transcript transcription prosodic POS tagging (task) tag pos tagger chunk accuracy tagging speech unknown tagset sequence treebank wsj fragment accuracy bracket pcfg probability np penn treebank head Lexicons (data) lexical lexicon item entry lex lexeme coverage associate derive substitution Constituent parsing (algorithm) clause constituent head relative coordination subject element position complement mark parse parser grammar chart parsing span tree syntactic stage Multilinguality (task) resource french spanish multilingual pivot german corpora italian dutch portuguese candidate rank selection ranking denote weight framework ranker probability combination Embeddings (algorithm) vector embedding matrix embed space dimension vec dimensional mikolov tensor Plan-based dialogue (task,algorithm) dialogue utterance act speaker plan turn goal belief conversation request Question answering (task) question answer passage question answere match paragraph trec reason factoid relevant Event extraction (task) event trigger mention extraction document ace attack argument entity relevant Grammars (algorithm) grammar derivation symbol terminal production nonterminal free cfg adjoin string Logical forms (algorithm) formula logic interpretation logical scope operator theory proposition predicate expression Knowledge base (data) entity mention wikipedia link person kb article document page title Information extraction (task) pattern seed extraction acquire acquisition bootstrappe web relationship discover match Applications (task) user tool module interface component support file format display design Disambiguation (task) interpretation ambiguity ambiguous processing preference strategy disambiguation attachment mechanism heuristic Graphs/AMR (algorithm) graph edge node vertex graphs connect amr weight propagation link Neural networks (algorithm) network layer neural cnn architecture rnn vector deep hide embedding Narratives (task) story genre book expert worker movie narrative human collect crowdsource Ontologies (algorithm) concept attribute hierarchy ontology taxonomy conceptual hypernym relation hierarchical link Prediction (task) predict prediction accuracy regression predictor error linear predictive variable effect Quantitative analysis (algorithm) frequency count probability distribution estimate occurrence corpora association statistical log Vision/multimodal (task) image visual video caption multimodal modality fusion textual human modal Parallel corpora (data) parallel bilingual monolingual corpora cross lingual keyphrase comparable translation resource extraction Neural MT (task,algorithm) decoder encoder nmt sequence neural attention bleu decode rnn vocabulary Recurrent neural networks (algorithm) lstm attention vector memory embed mechanism embedding weight layer encode Complexity analysis (task) cost memory speed index fast run bit store key efficient Opinion mining (task) review aspect product rating restaurant opinion customer rationale hotel service Social media (data) tweet twitter social media user twitt message hashtag detection post microblog Transliteration (task) character chinese transliteration oov hindi unknown accuracy char urdu ctb Dictionaries (data) dictionary definition code entry link cod dictionarie analogy database bank Relation extraction (task) relation extraction triple relational tuple open express relationship distant supervision rel Historical linguistics (task) change family lemma cognate russian czech linguistic historical distance swedish Wordnet/disambiguation (task,algorithm) sense wordnet sens disambiguation synset wsd gloss disambiguate resource ambiguous Dependency parsing (algorithm) search transition stack beam prune action shift greedy partial configuration Information retrieval (algorithm) query search web retrieval document page relevant retrieve relevance engine Supertagging (algorithm) category ccg np derivation supertag composition lexical ambiguity supertagger ccgbank Asian languages (task) japanese expression korean bunsetsu particle accuracy wo marker element noun Classification (algorithm) class classification classifier accuracy classify svm binary decision classifi combination Sequence analysis (algorithm) sequence local position global distance length chain sequential gap permutation Frame semantics (algorithm) frame slot schema filler framenet fill intent element role slu Dynamic programming (algorithm) path factor ij lattice tuple cache length denote space dynamic News articles (data) article news company year political country day people issue market Scene description (task) object description property expression µi µi reference scene referent location spatial  , oo, ooooo, uooo, oo, uu, uuuu precision, recall, match, detection, filter, extraction, threshold, detect, confidence, identification keyword, title, conference, computational linguistic, page, proceeding, tutorial, university, year, processing german, read, incremental, reading, die, prime, processing, der, surprisal, field