An Exploration of Arbitrary-Order Sequence Labeling via Energy-Based Inference Networks

Lifu Tu, Tianyu Liu, Kevin Gimpel


Abstract
Many tasks in natural language processing involve predicting structured outputs, e.g., sequence labeling, semantic role labeling, parsing, and machine translation. Researchers are increasingly applying deep representation learning to these problems, but the structured component of these approaches is usually quite simplistic. In this work, we propose several high-order energy terms to capture complex dependencies among labels in sequence labeling, including several that consider the entire label sequence. We use neural parameterizations for these energy terms, drawing from convolutional, recurrent, and self-attention networks. We use the framework of learning energy-based inference networks (Tu and Gimpel, 2018) for dealing with the difficulties of training and inference with such models. We empirically demonstrate that this approach achieves substantial improvement using a variety of high-order energy terms on four sequence labeling tasks, while having the same decoding speed as simple, local classifiers. We also find high-order energies to help in noisy data conditions.
Anthology ID:
2020.emnlp-main.449
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Editors:
Bonnie Webber, Trevor Cohn, Yulan He, Yang Liu
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5569–5582
Language:
URL:
https://aclanthology.org/2020.emnlp-main.449
DOI:
10.18653/v1/2020.emnlp-main.449
Bibkey:
Cite (ACL):
Lifu Tu, Tianyu Liu, and Kevin Gimpel. 2020. An Exploration of Arbitrary-Order Sequence Labeling via Energy-Based Inference Networks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5569–5582, Online. Association for Computational Linguistics.
Cite (Informal):
An Exploration of Arbitrary-Order Sequence Labeling via Energy-Based Inference Networks (Tu et al., EMNLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.emnlp-main.449.pdf
Video:
 https://slideslive.com/38939161
Code
 tyliupku/Arbitrary-Order-Infnet