How to Train good Word Embeddings for Biomedical NLP
Billy Chiu | Gamal Crichton | Anna Korhonen | Sampo Pyysalo
Proceedings of the 15th Workshop on Biomedical Natural Language Processing
Attending to Characters in Neural Sequence Labeling Models
Marek Rei | Gamal Crichton | Sampo Pyysalo
Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers
Sequence labeling architectures use word embeddings for capturing similarity, but suffer when handling previously unseen or rare words. We investigate character-level extensions to such models and propose a novel architecture for combining alternative word representations. By using an attention mechanism, the model is able to dynamically decide how much information to use from a word- or character-level component. We evaluated different architectures on a range of sequence labeling datasets, and character-level extensions were found to improve performance on every benchmark. In addition, the proposed attention-based architecture delivered the best results even with a smaller number of trainable parameters.