Language Model Metrics and Procrustes Analysis for Improved Vector Transformation of NLP Embeddings

Thomas Conley, Jugal Kalita


Abstract
Artificial Neural networks are mathematical models at their core. This truism presents some fundamental difficulty when networks are tasked with Natural Language Processing. A key problem lies in measuring the similarity or distance among vectors in NLP embedding space, since the mathematical concept of distance does not always agree with the linguistic concept. We suggest that the best way to measure linguistic distance among vectors is by employing the Language Model (LM) that created them. We introduce Language Model Distance (LMD) for measuring accuracy of vector transformations based on the Distributional Hypothesis ( LMD Accuracy ). We show the efficacy of this metric by applying it to a simple neural network learning the Procrustes algorithm for bilingual word mapping.
Anthology ID:
2020.icon-main.22
Volume:
Proceedings of the 17th International Conference on Natural Language Processing (ICON)
Month:
December
Year:
2020
Address:
Indian Institute of Technology Patna, Patna, India
Editors:
Pushpak Bhattacharyya, Dipti Misra Sharma, Rajeev Sangal
Venue:
ICON
SIG:
Publisher:
NLP Association of India (NLPAI)
Note:
Pages:
170–174
Language:
URL:
https://aclanthology.org/2020.icon-main.22
DOI:
Bibkey:
Cite (ACL):
Thomas Conley and Jugal Kalita. 2020. Language Model Metrics and Procrustes Analysis for Improved Vector Transformation of NLP Embeddings. In Proceedings of the 17th International Conference on Natural Language Processing (ICON), pages 170–174, Indian Institute of Technology Patna, Patna, India. NLP Association of India (NLPAI).
Cite (Informal):
Language Model Metrics and Procrustes Analysis for Improved Vector Transformation of NLP Embeddings (Conley & Kalita, ICON 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.icon-main.22.pdf