Identification, Interpretability, and Bayesian Word Embeddings

Adam Lauretig


Abstract
Social scientists have recently turned to analyzing text using tools from natural language processing like word embeddings to measure concepts like ideology, bias, and affinity. However, word embeddings are difficult to use in the regression framework familiar to social scientists: embeddings are are neither identified, nor directly interpretable. I offer two advances on standard embedding models to remedy these problems. First, I develop Bayesian Word Embeddings with Automatic Relevance Determination priors, relaxing the assumption that all embedding dimensions have equal weight. Second, I apply work identifying latent variable models to anchor embeddings, identifying them, and making them interpretable and usable in a regression. I then apply this model and anchoring approach to two cases, the shift in internationalist rhetoric in the American presidents’ inaugural addresses, and the relationship between bellicosity in American foreign policy decision-makers’ deliberations. I find that inaugural addresses became less internationalist after 1945, which goes against the conventional wisdom, and that an increase in bellicosity is associated with an increase in hostile actions by the United States, showing that elite deliberations are not cheap talk, and helping confirm the validity of the model.
Anthology ID:
W19-2102
Volume:
Proceedings of the Third Workshop on Natural Language Processing and Computational Social Science
Month:
June
Year:
2019
Address:
Minneapolis, Minnesota
Editors:
Svitlana Volkova, David Jurgens, Dirk Hovy, David Bamman, Oren Tsur
Venue:
NLP+CSS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7–17
Language:
URL:
https://aclanthology.org/W19-2102
DOI:
10.18653/v1/W19-2102
Bibkey:
Cite (ACL):
Adam Lauretig. 2019. Identification, Interpretability, and Bayesian Word Embeddings. In Proceedings of the Third Workshop on Natural Language Processing and Computational Social Science, pages 7–17, Minneapolis, Minnesota. Association for Computational Linguistics.
Cite (Informal):
Identification, Interpretability, and Bayesian Word Embeddings (Lauretig, NLP+CSS 2019)
Copy Citation:
PDF:
https://aclanthology.org/W19-2102.pdf
Code
 adamlauretig/bwe +  additional community code