How Many Layers and Why? An Analysis of the Model Depth in Transformers

Antoine Simoulin, Benoit Crabbé


Abstract
In this study, we investigate the role of the multiple layers in deep transformer models. We design a variant of Albert that dynamically adapts the number of layers for each token of the input. The key specificity of Albert is that weights are tied across layers. Therefore, the stack of encoder layers iteratively repeats the application of the same transformation function on the input. We interpret the repetition of this application as an iterative process where the token contextualized representations are progressively refined. We analyze this process at the token level during pre-training, fine-tuning, and inference. We show that tokens do not require the same amount of iterations and that difficult or crucial tokens for the task are subject to more iterations.
Anthology ID:
2021.acl-srw.23
Volume:
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop
Month:
August
Year:
2021
Address:
Online
Editors:
Jad Kabbara, Haitao Lin, Amandalynne Paullada, Jannis Vamvas
Venues:
ACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
221–228
Language:
URL:
https://aclanthology.org/2021.acl-srw.23
DOI:
10.18653/v1/2021.acl-srw.23
Bibkey:
Cite (ACL):
Antoine Simoulin and Benoit Crabbé. 2021. How Many Layers and Why? An Analysis of the Model Depth in Transformers. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop, pages 221–228, Online. Association for Computational Linguistics.
Cite (Informal):
How Many Layers and Why? An Analysis of the Model Depth in Transformers (Simoulin & Crabbé, ACL-IJCNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.acl-srw.23.pdf
Data
GLUE