Changing the Basis of Contextual Representations with Explicit Semantics

Tamás Ficsor, Gábor Berend


Abstract
The application of transformer-based contextual representations has became a de facto solution for solving complex NLP tasks. Despite their successes, such representations are arguably opaque as their latent dimensions are not directly interpretable. To alleviate this limitation of contextual representations, we devise such an algorithm where the output representation expresses human-interpretable information of each dimension. We achieve this by constructing a transformation matrix based on the semantic content of the embedding space and predefined semantic categories using Hellinger distance. We evaluate our inferred representations on supersense prediction task. Our experiments reveal that the interpretable nature of transformed contextual representations makes it possible to accurately predict the supersense category of a word by simply looking for its transformed coordinate with the largest coefficient. We quantify the effects of our proposed transformation when applied over traditional dense contextual embeddings. We additionally investigate and report consistent improvements for the integration of sparse contextual word representations into our proposed algorithm.
Anthology ID:
2021.acl-srw.25
Volume:
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop
Month:
August
Year:
2021
Address:
Online
Editors:
Jad Kabbara, Haitao Lin, Amandalynne Paullada, Jannis Vamvas
Venues:
ACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
235–247
Language:
URL:
https://aclanthology.org/2021.acl-srw.25
DOI:
10.18653/v1/2021.acl-srw.25
Bibkey:
Cite (ACL):
Tamás Ficsor and Gábor Berend. 2021. Changing the Basis of Contextual Representations with Explicit Semantics. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop, pages 235–247, Online. Association for Computational Linguistics.
Cite (Informal):
Changing the Basis of Contextual Representations with Explicit Semantics (Ficsor & Berend, ACL-IJCNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.acl-srw.25.pdf
Optional supplementary material:
 2021.acl-srw.25.OptionalSupplementaryMaterial.zip
Video:
 https://aclanthology.org/2021.acl-srw.25.mp4
Code
 ficstamas/word_embedding_interpretability