An Empirical Investigation of Beam-Aware Training in Supertagging

Renato Negrinho, Matthew R. Gormley, Geoff Gordon


Abstract
Structured prediction is often approached by training a locally normalized model with maximum likelihood and decoding approximately with beam search. This approach leads to mismatches as, during training, the model is not exposed to its mistakes and does not use beam search. Beam-aware training aims to address these problems, but unfortunately, it is not yet widely used due to a lack of understanding about how it impacts performance, when it is most useful, and whether it is stable. Recently, Negrinho et al. (2018) proposed a meta-algorithm that captures beam-aware training algorithms and suggests new ones, but unfortunately did not provide empirical results. In this paper, we begin an empirical investigation: we train the supertagging model of Vaswani et al. (2018) and a simpler model with instantiations of the meta-algorithm. We explore the influence of various design choices and make recommendations for choosing them. We observe that beam-aware training improves performance for both models, with large improvements for the simpler model which must effectively manage uncertainty during decoding. Our results suggest that a model must be learned with search to maximize its effectiveness.
Anthology ID:
2020.findings-emnlp.406
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2020
Month:
November
Year:
2020
Address:
Online
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4534–4542
Language:
URL:
https://aclanthology.org/2020.findings-emnlp.406
DOI:
10.18653/v1/2020.findings-emnlp.406
Bibkey:
Cite (ACL):
Renato Negrinho, Matthew R. Gormley, and Geoff Gordon. 2020. An Empirical Investigation of Beam-Aware Training in Supertagging. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4534–4542, Online. Association for Computational Linguistics.
Cite (Informal):
An Empirical Investigation of Beam-Aware Training in Supertagging (Negrinho et al., Findings 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.findings-emnlp.406.pdf
Optional supplementary material:
 2020.findings-emnlp.406.OptionalSupplementaryMaterial.zip
Video:
 https://slideslive.com/38940654
Code
 negrinho/beam_learn_supertagging