Ikuya Yamada1 2 3 Hideaki Takeda2 Yoshiyasu Takefuji3 ikuya@ousia.jp takeda@nii.ac.jp takefuji@sfc.keio.ac.jp 1Studio Ousia, 4489-105-221 Endo, Fujisawa, Kanagawa, Japan 2National Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda, Tokyo, Japan 3Keio University, 5322 Endo, Fujisawa, Kanagawa, Japan Abstract In this paper, we describe our approach for Named Entity Recognition in Twitter, a shared task for ACL 2015 Workshop on Noisy User-generated Text (Baldwin et al., 2015). Because of the noisy, short, and colloquial nature of Twitter, the per- formance of Named Entity Recognition (NER) degrades significantly. To address this problem, we propose a novel method to enhance the performance of the Twitter NER task by using Entity Linking which is a method for detecting entity mentions in text and resolving them to correspond- ing entries in knowledge bases such as Wikipedia. Our method is based on super- vised machine-learning and uses the high- quality knowledge obtained from several open knowledge bases. In comparison with the other systems proposed for this shared task, our method achieved the best performance. 1 Introduction Named Entity Recognition (NER) refers to the task of identifying mentions of entities (e.g., per- sons, locations, organizations) within text. Be- cause of the noisy, short, and colloquial nature of Twitter messages (or tweets), the performance of standard NER software significantly suffers. For example, Derczynski et al. (Derczynski et al., 2015) recently demonstrated that the performance of various state-of-the-art NER software (e.g., Stanford NER and ANNIE) is typically lower than 50% F11 for tweets. Entity Linking (EL) refers to the task of de- tecting textual entity mentions and linking them to corresponding entries within knowledge bases (e.g., Wikipedia, DBpedia (Auer et al., 2007), 1The harmonic mean of precision and recall. Freebase (Bollacker et al., 2008)). Because of the recent emergence of large online knowledge bases (KB), EL has recently gained significant at- tention. It is evident that the performance of EL also degrades when analyzing tweets (Derczynski et al., 2015; Meij et al., 2012). However, Guo et al. (Guo et al., 2013) recently revealed that the main failures of Twitter EL are caused while detecting entity mentions from text, because ex- isting EL methods usually address the mention detection task by using external NER software whose performance is unreliable when processing tweets. Consequently, several approaches (Guo et al., 2013; Yamada et al., 2015) have been proposed with enhanced abilities that address the task in an end-to-end manner without completely depending on NER software. The main objective of this study is to investigate the possibility of enhancing the performance of Twitter NER by using an end-to-end EL. Although EL is typically performed after NER in most of the existing methods, our approach performs EL before NER and uses the EL results to enhance the NER performance. Resolving the entity mentions to the KB entries enables us to use the high-quality knowledge in KB for enhancing the NER perfor- mance. This knowledge includes things such as the popularity of the entity, the classes of the en- tity, and the likelihood that the entity appears in the given context. We begin by briefly introducing our end-to-end EL method that specifically focuses on tweets. Our EL method is based on supervised machine- learning and addresses the task in an end-to-end manner. It considers every possible n-gram as a candidate entity mention and detects the mention with a corresponding link to a KB entry if the men- tion exists in the KB. Furthermore, it can handle mentions that appear as irregular forms (e.g., mis- spellings, abbreviations, acronyms) using several approximate string matching algorithms. 136 Proceedings of the ACL 2015 Workshop on Noisy User-generated Text, pages 136–140, Beijing, China, July 31, 2015. c�2015 Association for Computational Linguistics The NER task is split into two separate sub- tasks: segmentation and classification. During segmentation, the entity mentions are detected from tweets. Then, the entity mentions are clas- sified into the predefined entity types. Both tasks involve supervised machine-learning with various features. For the segmentation task, we use data obtained from the KB of the corresponding entity men- tion detected by the EL and the output of a NER software as the main machine-learning features. Furthermore, we include several common features used in traditional NER methods. For the classification task, the following three types of features are used as primary features: 1) the KB types of the entity detected by the EL, 2) the entity types detected by the NER software, and 3) the vector representation of the entity mention derived from word embeddings. The entity’s KB types are extracted from the corresponding entries in DBpedia and Freebase. Furthermore, the vec- tor representation of the entity mention is derived using GloVe word embeddings (Pennington et al., 2014). To train and evaluate our system, we used the dataset given by the Named Entity Recognition in Twitter shared task. Our proposed method signifi- cantly outperformed the second ranked system by a wide margin; 10.3% F1 at the segmentation task, and 5.0% F1 at the end-to-end (both the segmen- tation and the classification) task. 2 The Proposed System 2.1 Preprocessing The system first assigns part-of-speech tags to the resulting tokens using ARK Twitter Part-of- Speech Tagger (Gimpel et al., 2011). It also to- kenizes Twitter hashtags using our enhanced im- plementation of the hashtag tokenization. 2.2 Entity Linking We formalize our EL task as follows: Given a tweet, our goal is to recognize a set of entity mentions (e.g., Obama, President Obama, Barack Obama) that appear in a tweet, and then resolve the mentions into entities (e.g., Barack Obama) in Wikipedia if they exist. Our EL system addresses the task using the fol- lowing two steps; mention candidate generation and mention detection and disambiguation. 2.2.1 Mention Candidate Generation Our system first generates a set of candidate en- tity mentions with the set of corresponding refer- ent entities. The system takes all the n-grams of n ≤ 10 and looks up each n-gram in a dictionary, treats an n-gram as a candidate mention if it exists in the dictionary, and finally, generates an output of pairs of mentions and their associated possible referent entities. Mention-Entity Dictionary: The system uses a mention-entity dictionary that maps a mention sur- face (e.g., apple) to the possible referent entities (e.g., Apple Inc., Apple (food)). The possible mention surfaces of an entity are extracted from the corresponding Wikipedia page title, the page titles of the Wikipedia pages that redirect to the page of the entity, and anchor texts in Wikipedia articles that point to the page of the entity. We constructed this dictionary using the January 2015 dump of Wikipedia. Approximate Candidate Generation: One major problem of the mention candidate gen- eration task is that many entity mentions in tweets cannot be detected because they appear as irregular forms (e.g., misspellings, abbreviations). In order to address this problem, we introduce the following three approximate string-matching methods to improve the ability of this task: • Fuzzy match searches the mention candidates that have text surfaces within a certain dis- tance of the surface of the n-gram measured by edit distance. • Approximate token search obtains mention candidates whose text surfaces have a signif- icant ratio of words in common with the sur- face of the n-gram. • Acronym search retrieves mention candidates with possible acronyms2 that include the sur- face of the n-gram. When using the above methods, we observed that the number of mention candidates becomes very large. To deal with this, we use a simple fil- tering method based on soft tf-idf (Cohen et al., 2003); we simply use only the mention candidates that have a similarity greater than a threshold mea- sured by the soft tf-idf. We use 0.9 as the threshold 2We generate acronyms by tokenizing the mention surface and simply taking the first characters of the resulting tokens. 137 because this achieves the best performance in our experiments of EL. 2.2.2 Mention Detection and Disambiguation Given a pair of a mention and its possible refer- ent entity, it needs to be determined if the possible referent entity is indeed the correct one for its as- sociated mention. In this system, we use a supervised machine- learning algorithm to assign a relevance score to each of the pairs and select the entity mention with the highest score. We use random forest as the machine-learning algorithm. Here, we use machine-learning features that are mostly identical to the method proposed previ- ously (Yamada et al., 2015). Basically, we use var- ious features that are commonly observed in EL studies and enhance the performance further by introducing two new features: 1) the entity pop- ularity knowledge extracted from Wikipedia page views3, and 2) the contextual similarity between the entity and the tweet measured by word embed- dings. 2.3 Named Entity Recognition We address the NER task by performing two sub- tasks: segmentation and classification. 2.3.1 Segmentation of Named Entities In this step, entity mentions are detected from tweets. We formalize this task as follows. Given an n-gram in a tweet, the goal of this task is assigning a binary label that represents whether the n-gram should be detected as an entity men- tion. Note that in order to enable the straight- forward integration of EL and this task, we for- malize this task as simply classifying n-grams instead of the commonly-used IOB labeling ap- proach (Ramshaw and Marcus, 1995). The basic strategy that we adopt here is to com- bine the output of NER software and the KB knowledge of the corresponding entity mention detected by the EL using supervised machine- learning. We again use random forest as the machine-learning algorithm. We use Stanford NER4 as the NER software that achieves relatively better performance in the Twit- ter NER task in a recent study (Derczynski et al., 3http://dumps.wikimedia.org/other/ pagecounts-raw/ 4http://nlp.stanford.edu/software/ CRF-NER.shtml 2015). Here, we adopt two models of Stanford NER to enhance the performance: 1) the standard three-class model which is included in the soft- ware and 2) a model that does not use capitaliza- tion as a feature, in order to deal with the unrelia- bility of capitalization in tweets. The results of the NER and the KB knowledge of the corresponding entity mention detected by the EL are used as the primary machine-learning features. We also include features that are tradi- tionally used in NER such as part-of-speech tags and the capitalization features. Furthermore, the ratio of the capitalized words in the tweet is also used as an indicator of the reliability of the capi- talization. The machine-learning features for this step in- clude: • EL relevance score*: The relevance score of the entity mention assigned by the previous EL step. • Link probability*: The probability of the en- tity mention appearing as an anchor text in Wikipedia. • Capitalization probability*: The probability of the entity mention being capitalized in Wikipedia. • The number of inbound links*: The number of inbound links of the corresponding entity in Wikipedia. • The average page view*: The average page view of the corresponding entity in Wikipedia. • NER span match: Binary values that repre- sent whether the n-gram is detected by NER models. • Part-of-speech tags: Part-of-speech tags of the previous, first, last, and next words of the n-gram. • Context capitalization: Binary values that represent whether the previous, first, last, and next words of the n-gram are capitalized. • Character length: The number of characters read in the surface of the n-gram. • Token length: The number of tokens read in the n-gram. Note that some features (marked with *) are based on an entity mention detected by EL, thus 138 these features can be missing if there is no corre- sponding entity mention detected by the EL. We also resolve overlaps of mentions by itera- tively selecting the longest entity mention from the beginning of a tweet. 2.3.2 Classification of Named Entities In this step, detected entity mentions are classified into the predefined types (i.e., person, geo-loc, facility, product, company, movie, sportsteam, musicartist, tvshow, and other) using supervised machine-learning. Here, linear support vector ma- chine is used as the machine-learning model. One main machine-learning feature of this step is the corresponding entity types retrieved from KBs. We obtain KB entity types from the corre- sponding entries in DBpedia5 and Freebase6. One problem in this step is that there are several entity mentions that cannot be detected by EL be- cause of various reasons (e.g., a non-existent entity in the KB, an error performing EL). In addition, some minor entities might not have entity types in the KBs. In order to deal with this problem, we first include the entity types predicted by Stanford NER as features. However, because the target en- tity types of our task do not directly correspond to the ones given in Stanford NER (i.e., location, person, and organization), the effectiveness of these features is obviously limited. Therefore, we introduce another type of feature based on word embeddings. For this, we use GloVe word embed- dings7 to calculate an average vector of vectors of words in n-gram text. We also include the relevance score assigned by the previous EL step that indicates the reliability of the KB entity types to the model. The number of words and the number of characters in the n- gram text are also included as features to enhance the expressiveness of our model even further. The machine-learning features for this step in- clude: • KB entity types: The entity types in KBs. The KBs used include DBpedia and Freebase. • NER detected type: The detected entity types of the NER model. As mentioned in Section 5http://mappings.dbpedia.org/server/ ontology/classes/ 6http://wiki.freebase.com/wiki/Type 7We use the 300-dimensional model generated using 840B tokens obtained from CommonCrawl corpus. http: //nlp.stanford.edu/projects/glove/ System Name Precision Recall F1 Our Method 72.20% 69.14% 70.63% NLANGP 67.74% 54.31% 60.29% USFD 63.81% 56.28% 59.81% multimedialab 62.93% 55.22% 58.82% nrc 62.13% 54.61% 58.13% Table 1: Performances of the proposed systems at segmenting entities 2.3.1, we use two different models of Stan- ford NER. • N-gram vector: The vector representation of the n-gram derived using the method ex- plained above and includes each dimension of the vector as a separate feature. • EL relevance score: The relevance score as- signed by the previous EL step. • Character length: The number of characters read in the n-gram text. • Token length: The number of tokens read in the n-gram. 3 Experiments 3.1 Experimental Setup To train our proposed EL method, we used the #Microposts 2015 EL dataset (Rizzo et al., 2015) that contains 3,998 tweets and 3,993 annotations of entities.8 The performance of our EL method using this particular dataset is reported in (Yamada et al., 2015). For this shared task, we trained and evaluated our proposed Twitter NER using the dataset pro- vided by the workshop.9 3.2 Results Table 1 shows the results of the segmentation task of the five top-ranking systems. Our pro- posed method significantly outperforms the sec- ond ranked method by 10.3% F1. The end-to-end results (both segmentation and classification tasks) of the five top-ranking sys- tems are shown in Table 2. Here, our method sig- nificantly outperforms the second ranked method by 5.0% F1. Table 3 also presents detailed scores broken down by entity types. 8We use the training and the dev set of the #Microposts 2015 dataset as the training data. 9We use the train, the dev, and the dev 2015 set for train- ing the NER model. 139 System Name Precision Recall F1 Our Method 57.66% 55.22% 56.41% NLANGP 63.62% 41.12% 51.40% nrc 53.24% 38.58% 44.74% multimedialab 49.52% 39.18% 43.75% USFD 45.72% 39.64% 42.46% Table 2: Performances of the proposed systems at both segmentation and classification tasks Entity Type Precision Recall F1 company 41.82% 58.97% 48.94% facility 50.00% 26.32% 34.48% geo-loc 57.59% 78.45% 66.42% movie 66.67% 40.00% 50.00% musicartist 70.00% 34.15% 45.90% other 47.06% 42.42% 44.62% person 70.97% 77.19% 73.95% product 34.78% 21.62% 26.67% sportsteam 66.67% 34.29% 45.28% tvshow 14.29% 50.00% 22.22% Table 3: Performance of our system at both seg- mentation and classification tasks broken down by entity types 4 Conclusions In this paper, we proposed a novel method for the Twitter NER task. We showed that the data retrieved from open knowledge bases (i.e., Wikipedia, DBpedia, Freebase) can be naturally leveraged to enhance NER using entity linking. Furthermore, this data appears to be highly effec- tive for both the segmentation and the classifica- tion tasks. References S¨oren Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. DBpedia: a nucleus for a web of open data. The Semantic Web, pages 722–735. Timothy Baldwin, Marie Catherine de Marneffe, Bo Han, Young-Bum Kim, Alan Ritter, and Wei Xu. 2015. Shared tasks of the 2015 workshop on noisy user-generated text: Twitter lexical normal- ization and named entity recognition. In Proceed- ings of the Workshop on Noisy User-generated Text (WNUT 2015). Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a col- laboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data (SIGMOD ’08), pages 1247–1250. William W. Cohen, Pradeep D. Ravikumar, and Stephen E. Fienberg. 2003. A comparison of string distance metrics for name-matching tasks. In Pro- ceedings of IJCAI-03 Workshop on Information In- tegration on the Web, pages 73–78. Leon Derczynski, Diana Maynard, Giuseppe Rizzo, Marieke van Erp, Genevieve Gorrell, Rapha¨el Troncy, Johann Petrak, and Kalina Bontcheva. 2015. Analysis of named entity recognition and linking for tweets. Information Processing & Man- agement, 51(2):32–49. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A Smith. 2011. Part-of-speech tagging for Twitter: annotation, features, and experiments. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (HLT ’11), pages 42–47. Stephen Guo, Ming-Wei Chang, and Emre Kiciman. 2013. To link or not to link? a study on end-to-end Tweet entity linking. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT ’13), pages 1020–1030. Edgar Meij, Wouter Weerkamp, and Maarten de Rijke. 2012. Adding semantics to microblog posts. In Pro- ceedings of the Fifth ACM International Conference on Web Search and Data Mining (WSDM ’12), pages 563–572. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Con- ference on Empirical Methods in Natural Language Processing (EMNLP ’14), pages 1532–1543. Lance A. Ramshaw and Mitchell P. Marcus. 1995. Text chunking using transformation-based learning. In Proceedings of ACL Third Workshop on Very Large Corpora, pages 82–94. Giuseppe Rizzo, Amparo Elizabeth Cano Basave, Bianca Pereira, Andrea Varga, Matthew Rowe, Mi- lan Stankovic, and Aba-Sah Dadzie. 2015. Making sense of microposts (#Microposts2015) named en- tity recognition and linking (NEEL) challenge. In 5th Workshop on Making Sense of Microposts (#Mi- croposts2015). Ikuya Yamada, Hideaki Takeda, and Yoshiyasu Take- fuji. 2015. An end-to-end entity linking approach for Tweets. In 5th Workshop on Making Sense of Microposts (#Microposts 2015). 140 2 3 Yoshiyasu ikuya@ousia.jp takeda@nii.ac.jp Ousia, 4489-105-221 Endo, Fujisawa, Kanagawa, Japan Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda, Tokyo, Japan University, 5322 Endo, Fujisawa, Kanagawa, Japan In this paper, we describe our approach Entity Recognition in a shared task for ACL 2015 Workshop on Noisy User-generated Text (Baldwin et al., 2015). Because of the noisy, short, and colloquial nature of Twitter, the performance of Named Entity Recognition (NER) degrades significantly. To address this problem, we propose a novel method to enhance the performance of the Twitter task by using Linking is a method for detecting entity mentions in text and resolving them to corresponding entries in knowledge bases such as Wikipedia. Our method is based on supervised machine-learning and uses the highquality knowledge obtained from several open knowledge bases. In comparison with the other systems proposed for this shared task, our method achieved the best performance. S¨oren Auer Christian Bizer Georgi Kobilarov Jens Lehmann Richard Cyganiak Zachary Ives 2007 722--735 of entities (e.g., persons, locations, organizations) within text. Because of the noisy, short, and colloquial nature of Twitter messages (or tweets), the performance of standard NER software significantly suffers. For example, Derczynski et al. (Derczynski et al., 2015) recently demonstrated that the performance of various state-of-the-art NER software (e.g., Stanford NER and ANNIE) is typically lower than 50% F11 for tweets. Entity Linking (EL) refers to the task of detecting textual entity mentions and linking them to corresponding entries within knowledge bases (e.g., Wikipedia, DBpedia (Auer et al., 2007), 1The harmonic mean of precision and recall. Freebase (Bollacker et al., 2008)). Because of the recent emergence of large online knowledge bases (KB), EL has recently gained significant attention. It is evident that the performance of EL also degrades when analyzing tweets (Derczynski et al., 2015; Meij et al., 2012). However, Guo et al. (Guo et al., 2013) recently revealed that the main failures of Twitter EL are caused while detecting entity mentions from text, because existing EL methods usually address the mention detection task by using external NER software whose performance is unreliab Auer, Bizer, Kobilarov, Lehmann, Cyganiak, Ives, 2007 S¨oren Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. DBpedia: a nucleus for a web of open data. The Semantic Web, pages 722–735. Timothy Baldwin Marie Catherine de Marneffe Bo Han Young-Bum Kim Alan Ritter Wei Xu 2015 In Proceedings of the Workshop on Noisy User-generated Text (WNUT Baldwin, de Marneffe, Han, Kim, Ritter, Xu, 2015 Timothy Baldwin, Marie Catherine de Marneffe, Bo Han, Young-Bum Kim, Alan Ritter, and Wei Xu. 2015. Shared tasks of the 2015 workshop on noisy user-generated text: Twitter lexical normalization and named entity recognition. In Proceedings of the Workshop on Noisy User-generated Text (WNUT 2015). Kurt Bollacker Colin Evans Praveen Paritosh Tim Sturge Jamie Taylor 2008 In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data (SIGMOD ’08), 1247--1250 e of the noisy, short, and colloquial nature of Twitter messages (or tweets), the performance of standard NER software significantly suffers. For example, Derczynski et al. (Derczynski et al., 2015) recently demonstrated that the performance of various state-of-the-art NER software (e.g., Stanford NER and ANNIE) is typically lower than 50% F11 for tweets. Entity Linking (EL) refers to the task of detecting textual entity mentions and linking them to corresponding entries within knowledge bases (e.g., Wikipedia, DBpedia (Auer et al., 2007), 1The harmonic mean of precision and recall. Freebase (Bollacker et al., 2008)). Because of the recent emergence of large online knowledge bases (KB), EL has recently gained significant attention. It is evident that the performance of EL also degrades when analyzing tweets (Derczynski et al., 2015; Meij et al., 2012). However, Guo et al. (Guo et al., 2013) recently revealed that the main failures of Twitter EL are caused while detecting entity mentions from text, because existing EL methods usually address the mention detection task by using external NER software whose performance is unreliable when processing tweets. Consequently, several approaches (Guo et al., 2013; Bollacker, Evans, Paritosh, Sturge, Taylor, 2008 Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data (SIGMOD ’08), pages 1247–1250. William W Cohen Pradeep D Ravikumar Stephen E Fienberg 2003 In Proceedings of IJCAI-03 Workshop on Information Integration on the Web, 73--78 of this task: • Fuzzy match searches the mention candidates that have text surfaces within a certain distance of the surface of the n-gram measured by edit distance. • Approximate token search obtains mention candidates whose text surfaces have a significant ratio of words in common with the surface of the n-gram. • Acronym search retrieves mention candidates with possible acronyms2 that include the surface of the n-gram. When using the above methods, we observed that the number of mention candidates becomes very large. To deal with this, we use a simple filtering method based on soft tf-idf (Cohen et al., 2003); we simply use only the mention candidates that have a similarity greater than a threshold measured by the soft tf-idf. We use 0.9 as the threshold 2We generate acronyms by tokenizing the mention surface and simply taking the first characters of the resulting tokens. 137 because this achieves the best performance in our experiments of EL. 2.2.2 Mention Detection and Disambiguation Given a pair of a mention and its possible referent entity, it needs to be determined if the possible referent entity is indeed the correct one for its associated mention. In this system, we use a supervised machine Cohen, Ravikumar, Fienberg, 2003 William W. Cohen, Pradeep D. Ravikumar, and Stephen E. Fienberg. 2003. A comparison of string distance metrics for name-matching tasks. In Proceedings of IJCAI-03 Workshop on Information Integration on the Web, pages 73–78. Leon Derczynski Diana Maynard Giuseppe Rizzo Marieke van Erp Genevieve Gorrell Johann Petrak Rapha¨el Troncy Kalina Bontcheva 2015 Information Processing & Management, 51 2 Derczynski, Maynard, Rizzo, van Erp, Gorrell, Rapha¨el Troncy, Bontcheva, 2015 Leon Derczynski, Diana Maynard, Giuseppe Rizzo, Marieke van Erp, Genevieve Gorrell, Rapha¨el Troncy, Johann Petrak, and Kalina Bontcheva. 2015. Analysis of named entity recognition and linking for tweets. Information Processing & Management, 51(2):32–49. Kevin Gimpel Nathan Schneider Brendan O’Connor Dipanjan Das Daniel Mills Jacob Eisenstein Michael Heilman Dani Yogatama Jeffrey Flanigan Noah A Smith 2011 In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (HLT ’11), 42--47 Gimpel, Schneider, O’Connor, Das, Mills, Eisenstein, Heilman, Yogatama, Flanigan, Smith, 2011 Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A Smith. 2011. Part-of-speech tagging for Twitter: annotation, features, and experiments. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (HLT ’11), pages 42–47. Stephen Guo Ming-Wei Chang Emre Kiciman 2013 In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT ’13), 1020--1030 g., Stanford NER and ANNIE) is typically lower than 50% F11 for tweets. Entity Linking (EL) refers to the task of detecting textual entity mentions and linking them to corresponding entries within knowledge bases (e.g., Wikipedia, DBpedia (Auer et al., 2007), 1The harmonic mean of precision and recall. Freebase (Bollacker et al., 2008)). Because of the recent emergence of large online knowledge bases (KB), EL has recently gained significant attention. It is evident that the performance of EL also degrades when analyzing tweets (Derczynski et al., 2015; Meij et al., 2012). However, Guo et al. (Guo et al., 2013) recently revealed that the main failures of Twitter EL are caused while detecting entity mentions from text, because existing EL methods usually address the mention detection task by using external NER software whose performance is unreliable when processing tweets. Consequently, several approaches (Guo et al., 2013; Yamada et al., 2015) have been proposed with enhanced abilities that address the task in an end-to-end manner without completely depending on NER software. The main objective of this study is to investigate the possibility of enhancing the performance of Twitter NER by using an e Guo, Chang, Kiciman, 2013 Stephen Guo, Ming-Wei Chang, and Emre Kiciman. 2013. To link or not to link? a study on end-to-end Tweet entity linking. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT ’13), pages 1020–1030. Edgar Meij Wouter Weerkamp Maarten de Rijke 2012 In Proceedings of the Fifth ACM International Conference on Web Search and Data Mining (WSDM ’12), 563--572 Meij, Weerkamp, de Rijke, 2012 Edgar Meij, Wouter Weerkamp, and Maarten de Rijke. 2012. Adding semantics to microblog posts. In Proceedings of the Fifth ACM International Conference on Web Search and Data Mining (WSDM ’12), pages 563–572. Jeffrey Pennington Richard Socher Christopher D Manning 2014 In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP ’14), 1532--1543 ware as the main machine-learning features. Furthermore, we include several common features used in traditional NER methods. For the classification task, the following three types of features are used as primary features: 1) the KB types of the entity detected by the EL, 2) the entity types detected by the NER software, and 3) the vector representation of the entity mention derived from word embeddings. The entity’s KB types are extracted from the corresponding entries in DBpedia and Freebase. Furthermore, the vector representation of the entity mention is derived using GloVe word embeddings (Pennington et al., 2014). To train and evaluate our system, we used the dataset given by the Named Entity Recognition in Twitter shared task. Our proposed method significantly outperformed the second ranked system by a wide margin; 10.3% F1 at the segmentation task, and 5.0% F1 at the end-to-end (both the segmentation and the classification) task. 2 The Proposed System 2.1 Preprocessing The system first assigns part-of-speech tags to the resulting tokens using ARK Twitter Part-ofSpeech Tagger (Gimpel et al., 2011). It also tokenizes Twitter hashtags using our enhanced implementation of the hashtag tokenization. 2.2 E Pennington, Socher, Manning, 2014 Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP ’14), pages 1532–1543. Lance A Ramshaw Mitchell P Marcus 1995 In Proceedings of ACL Third Workshop on Very Large Corpora, 82--94 word embeddings. 2.3 Named Entity Recognition We address the NER task by performing two subtasks: segmentation and classification. 2.3.1 Segmentation of Named Entities In this step, entity mentions are detected from tweets. We formalize this task as follows. Given an n-gram in a tweet, the goal of this task is assigning a binary label that represents whether the n-gram should be detected as an entity mention. Note that in order to enable the straightforward integration of EL and this task, we formalize this task as simply classifying n-grams instead of the commonly-used IOB labeling approach (Ramshaw and Marcus, 1995). The basic strategy that we adopt here is to combine the output of NER software and the KB knowledge of the corresponding entity mention detected by the EL using supervised machinelearning. We again use random forest as the machine-learning algorithm. We use Stanford NER4 as the NER software that achieves relatively better performance in the Twitter NER task in a recent study (Derczynski et al., 3http://dumps.wikimedia.org/other/ pagecounts-raw/ 4http://nlp.stanford.edu/software/ CRF-NER.shtml 2015). Here, we adopt two models of Stanford NER to enhance the performance: 1) the standard three-c Ramshaw, Marcus, 1995 Lance A. Ramshaw and Mitchell P. Marcus. 1995. Text chunking using transformation-based learning. In Proceedings of ACL Third Workshop on Very Large Corpora, pages 82–94. Giuseppe Rizzo Amparo Elizabeth Cano Basave Bianca Pereira Andrea Varga Matthew Rowe Milan Stankovic Aba-Sah Dadzie 2015 In 5th Workshop on Making Sense of Microposts (#Microposts2015). % Table 1: Performances of the proposed systems at segmenting entities 2.3.1, we use two different models of Stanford NER. • N-gram vector: The vector representation of the n-gram derived using the method explained above and includes each dimension of the vector as a separate feature. • EL relevance score: The relevance score assigned by the previous EL step. • Character length: The number of characters read in the n-gram text. • Token length: The number of tokens read in the n-gram. 3 Experiments 3.1 Experimental Setup To train our proposed EL method, we used the #Microposts 2015 EL dataset (Rizzo et al., 2015) that contains 3,998 tweets and 3,993 annotations of entities.8 The performance of our EL method using this particular dataset is reported in (Yamada et al., 2015). For this shared task, we trained and evaluated our proposed Twitter NER using the dataset provided by the workshop.9 3.2 Results Table 1 shows the results of the segmentation task of the five top-ranking systems. Our proposed method significantly outperforms the second ranked method by 10.3% F1. The end-to-end results (both segmentation and classification tasks) of the five top-ranking systems are shown in Table 2. Here, our method Rizzo, Basave, Pereira, Varga, Rowe, Stankovic, Dadzie, 2015 Giuseppe Rizzo, Amparo Elizabeth Cano Basave, Bianca Pereira, Andrea Varga, Matthew Rowe, Milan Stankovic, and Aba-Sah Dadzie. 2015. Making sense of microposts (#Microposts2015) named entity recognition and linking (NEEL) challenge. In 5th Workshop on Making Sense of Microposts (#Microposts2015). Ikuya Yamada Hideaki Takeda Yoshiyasu Takefuji 2015 In 5th Workshop on Making Sense of Microposts (#Microposts ). Because of the recent emergence of large online knowledge bases (KB), EL has recently gained significant attention. It is evident that the performance of EL also degrades when analyzing tweets (Derczynski et al., 2015; Meij et al., 2012). However, Guo et al. (Guo et al., 2013) recently revealed that the main failures of Twitter EL are caused while detecting entity mentions from text, because existing EL methods usually address the mention detection task by using external NER software whose performance is unreliable when processing tweets. Consequently, several approaches (Guo et al., 2013; Yamada et al., 2015) have been proposed with enhanced abilities that address the task in an end-to-end manner without completely depending on NER software. The main objective of this study is to investigate the possibility of enhancing the performance of Twitter NER by using an end-to-end EL. Although EL is typically performed after NER in most of the existing methods, our approach performs EL before NER and uses the EL results to enhance the NER performance. Resolving the entity mentions to the KB entries enables us to use the high-quality knowledge in KB for enhancing the NER performance. This knowledge include se this achieves the best performance in our experiments of EL. 2.2.2 Mention Detection and Disambiguation Given a pair of a mention and its possible referent entity, it needs to be determined if the possible referent entity is indeed the correct one for its associated mention. In this system, we use a supervised machinelearning algorithm to assign a relevance score to each of the pairs and select the entity mention with the highest score. We use random forest as the machine-learning algorithm. Here, we use machine-learning features that are mostly identical to the method proposed previously (Yamada et al., 2015). Basically, we use various features that are commonly observed in EL studies and enhance the performance further by introducing two new features: 1) the entity popularity knowledge extracted from Wikipedia page views3, and 2) the contextual similarity between the entity and the tweet measured by word embeddings. 2.3 Named Entity Recognition We address the NER task by performing two subtasks: segmentation and classification. 2.3.1 Segmentation of Named Entities In this step, entity mentions are detected from tweets. We formalize this task as follows. Given an n-gram in a tweet, the goal of thi ion of the n-gram derived using the method explained above and includes each dimension of the vector as a separate feature. • EL relevance score: The relevance score assigned by the previous EL step. • Character length: The number of characters read in the n-gram text. • Token length: The number of tokens read in the n-gram. 3 Experiments 3.1 Experimental Setup To train our proposed EL method, we used the #Microposts 2015 EL dataset (Rizzo et al., 2015) that contains 3,998 tweets and 3,993 annotations of entities.8 The performance of our EL method using this particular dataset is reported in (Yamada et al., 2015). For this shared task, we trained and evaluated our proposed Twitter NER using the dataset provided by the workshop.9 3.2 Results Table 1 shows the results of the segmentation task of the five top-ranking systems. Our proposed method significantly outperforms the second ranked method by 10.3% F1. The end-to-end results (both segmentation and classification tasks) of the five top-ranking systems are shown in Table 2. Here, our method significantly outperforms the second ranked method by 5.0% F1. Table 3 also presents detailed scores broken down by entity types. 8We use the training and the dev Yamada, Takeda, Takefuji, 2015 Ikuya Yamada, Hideaki Takeda, and Yoshiyasu Takefuji. 2015. An end-to-end entity linking approach for Tweets. In 5th Workshop on Making Sense of Microposts (#Microposts 2015).