Training for Gibbs Sampling on Conditional Random Fields with Neural Scoring Factors

Sida Gao, Matthew R. Gormley


Abstract
Most recent improvements in NLP come from changes to the neural network architectures modeling the text input. Yet, state-of-the-art models often rely on simple approaches to model the label space, e.g. bigram Conditional Random Fields (CRFs) in sequence tagging. More expressive graphical models are rarely used due to their prohibitive computational cost. In this work, we present an approach for efficiently training and decoding hybrids of graphical models and neural networks based on Gibbs sampling. Our approach is the natural adaptation of SampleRank (Wick et al., 2011) to neural models, and is widely applicable to tasks beyond sequence tagging. We apply our approach to named entity recognition and present a neural skip-chain CRF model, for which exact inference is impractical. The skip-chain model improves over a strong baseline on three languages from CoNLL-02/03. We obtain new state-of-the-art results on Dutch.
Anthology ID:
2020.emnlp-main.406
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Editors:
Bonnie Webber, Trevor Cohn, Yulan He, Yang Liu
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4999–5011
Language:
URL:
https://aclanthology.org/2020.emnlp-main.406
DOI:
10.18653/v1/2020.emnlp-main.406
Bibkey:
Cite (ACL):
Sida Gao and Matthew R. Gormley. 2020. Training for Gibbs Sampling on Conditional Random Fields with Neural Scoring Factors. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4999–5011, Online. Association for Computational Linguistics.
Cite (Informal):
Training for Gibbs Sampling on Conditional Random Fields with Neural Scoring Factors (Gao & Gormley, EMNLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.emnlp-main.406.pdf
Video:
 https://slideslive.com/38939315
Code
 GaoSida/Neural-SampleRank
Data
CoNLL 2003