Non-Autoregressive Sequence Generation

Jiatao Gu, Xu Tan


Abstract
Non-autoregressive sequence generation (NAR) attempts to generate the entire or partial output sequences in parallel to speed up the generation process and avoid potential issues (e.g., label bias, exposure bias) in autoregressive generation. While it has received much research attention and has been applied in many sequence generation tasks in natural language and speech, naive NAR models still face many challenges to close the performance gap between state-of-the-art autoregressive models because of a lack of modeling power. In this tutorial, we will provide a thorough introduction and review of non-autoregressive sequence generation, in four sections: 1) Background, which covers the motivation of NAR generation, the problem definition, the evaluation protocol, and the comparison with standard autoregressive generation approaches. 2) Method, which includes different aspects: model architecture, objective function, training data, learning paradigm, and additional inference tricks. 3) Application, which covers different tasks in text and speech generation, and some advanced topics in applications. 4) Conclusion, in which we describe several research challenges and discuss the potential future research directions. We hope this tutorial can serve both academic researchers and industry practitioners working on non-autoregressive sequence generation.
Anthology ID:
2022.acl-tutorials.4
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Luciana Benotti, Naoaki Okazaki, Yves Scherrer, Marcos Zampieri
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
21–27
Language:
URL:
https://aclanthology.org/2022.acl-tutorials.4
DOI:
10.18653/v1/2022.acl-tutorials.4
Bibkey:
Cite (ACL):
Jiatao Gu and Xu Tan. 2022. Non-Autoregressive Sequence Generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 21–27, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Non-Autoregressive Sequence Generation (Gu & Tan, ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-tutorials.4.pdf