If beam search is the answer, what was the question?

Clara Meister, Ryan Cotterell, Tim Vieira


Abstract
Quite surprisingly, exact maximum a posteriori (MAP) decoding of neural language generators frequently leads to low-quality results. Rather, most state-of-the-art results on language generation tasks are attained using beam search despite its overwhelmingly high search error rate. This implies that the MAP objective alone does not express the properties we desire in text, which merits the question: if beam search is the answer, what was the question? We frame beam search as the exact solution to a different decoding objective in order to gain insights into why high probability under a model alone may not indicate adequacy. We find that beam search enforces uniform information density in text, a property motivated by cognitive science. We suggest a set of decoding objectives that explicitly enforce this property and find that exact decoding with these objectives alleviates the problems encountered when decoding poorly calibrated language generation models. Additionally, we analyze the text produced using various decoding strategies and see that, in our neural machine translation experiments, the extent to which this property is adhered to strongly correlates with BLEU.
Anthology ID:
2020.emnlp-main.170
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2173–2185
Language:
URL:
https://aclanthology.org/2020.emnlp-main.170
DOI:
10.18653/v1/2020.emnlp-main.170
Award:
 Honorable Mention Paper
Bibkey:
Cite (ACL):
Clara Meister, Ryan Cotterell, and Tim Vieira. 2020. If beam search is the answer, what was the question?. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2173–2185, Online. Association for Computational Linguistics.
Cite (Informal):
If beam search is the answer, what was the question? (Meister et al., EMNLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.emnlp-main.170.pdf
Video:
 https://slideslive.com/38938891
Code
 rycolab/uid-decoding
Data
WMT 2014