@inproceedings{narang-etal-2021-transformer,
    title = "Do Transformer Modifications Transfer Across Implementations and Applications?",
    author = "Narang, Sharan  and
      Chung, Hyung Won  and
      Tay, Yi  and
      Fedus, Liam  and
      Fevry, Thibault  and
      Matena, Michael  and
      Malkan, Karishma  and
      Fiedel, Noah  and
      Shazeer, Noam  and
      Lan, Zhenzhong  and
      Zhou, Yanqi  and
      Li, Wei  and
      Ding, Nan  and
      Marcus, Jake  and
      Roberts, Adam  and
      Raffel, Colin",
    booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2021",
    address = "Online and Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.emnlp-main.465",
    doi = "10.18653/v1/2021.emnlp-main.465",
    pages = "5758--5773",
    abstract = "The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread adoption. In this paper, we comprehensively evaluate many of these modifications in a shared experimental setting that covers most of the common uses of the Transformer in natural language processing. Surprisingly, we find that most modifications do not meaningfully improve performance. Furthermore, most of the Transformer variants we found beneficial were either developed in the same codebase that we used or are relatively minor changes. We conjecture that performance improvements may strongly depend on implementation details and correspondingly make some recommendations for improving the generality of experimental results.",
}