Leveraging Relaxed Equilibrium by Lazy Transition for Sequence Modeling

Xi Ai, Bin Fang


Abstract
In sequence modeling, certain tokens are usually less ambiguous than others, and representations of these tokens require fewer refinements for disambiguation. However, given the nature of attention-based models like Transformer and UT (universal transformer), all tokens are equally processed towards depth. Inspired by the equilibrium phenomenon, we present a lazy transition, a mechanism to adjust the significance of iterative refinements for each token representation. Our lazy transition is deployed on top of UT to build LT (lazy transformer), where all tokens are processed unequally towards depth. Eventually, LT is encouraged to oscillate around a relaxed equilibrium. Our experiments show that LT outperforms baseline models on several tasks of machine translation, pre-training, Learning to Execute, and LAMBADA.
Anthology ID:
2022.acl-long.208
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2904–2924
Language:
URL:
https://aclanthology.org/2022.acl-long.208
DOI:
10.18653/v1/2022.acl-long.208
Bibkey:
Cite (ACL):
Xi Ai and Bin Fang. 2022. Leveraging Relaxed Equilibrium by Lazy Transition for Sequence Modeling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2904–2924, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Leveraging Relaxed Equilibrium by Lazy Transition for Sequence Modeling (Ai & Fang, ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-long.208.pdf
Software:
 2022.acl-long.208.software.zip
Data
LAMBADAMultiNLI