%0 Conference Proceedings %T On the differences between BERT and MT encoder spaces and how to address them in translation tasks %A Vázquez, Raúl %A Celikkanat, Hande %A Creutz, Mathias %A Tiedemann, Jörg %Y Kabbara, Jad %Y Lin, Haitao %Y Paullada, Amandalynne %Y Vamvas, Jannis %S Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop %D 2021 %8 August %I Association for Computational Linguistics %C Online %F vazquez-etal-2021-differences %X Various studies show that pretrained language models such as BERT cannot straightforwardly replace encoders in neural machine translation despite their enormous success in other tasks. This is even more astonishing considering the similarities between the architectures. This paper sheds some light on the embedding spaces they create, using average cosine similarity, contextuality metrics and measures for representational similarity for comparison, revealing that BERT and NMT encoder representations look significantly different from one another. In order to address this issue, we propose a supervised transformation from one into the other using explicit alignment and fine-tuning. Our results demonstrate the need for such a transformation to improve the applicability of BERT in MT. %R 10.18653/v1/2021.acl-srw.35 %U https://aclanthology.org/2021.acl-srw.35 %U https://doi.org/10.18653/v1/2021.acl-srw.35 %P 337-347