Topic Model or Topic Twaddle? Re-evaluating Semantic Interpretability Measures

Caitlin Doogan, Wray Buntine


Abstract
When developing topic models, a critical question that should be asked is: How well will this model work in an applied setting? Because standard performance evaluation of topic interpretability uses automated measures modeled on human evaluation tests that are dissimilar to applied usage, these models’ generalizability remains in question. In this paper, we probe the issue of validity in topic model evaluation and assess how informative coherence measures are for specialized collections used in an applied setting. Informed by the literature, we propose four understandings of interpretability. We evaluate these using a novel experimental framework reflective of varied applied settings, including human evaluations using open labeling, typical of applied research. These evaluations show that for some specialized collections, standard coherence measures may not inform the most appropriate topic model or the optimal number of topics, and current interpretability performance validation methods are challenged as a means to confirm model quality in the absence of ground truth data.
Anthology ID:
2021.naacl-main.300
Volume:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
June
Year:
2021
Address:
Online
Editors:
Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, Yichao Zhou
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3824–3848
Language:
URL:
https://aclanthology.org/2021.naacl-main.300
DOI:
10.18653/v1/2021.naacl-main.300
Bibkey:
Cite (ACL):
Caitlin Doogan and Wray Buntine. 2021. Topic Model or Topic Twaddle? Re-evaluating Semantic Interpretability Measures. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3824–3848, Online. Association for Computational Linguistics.
Cite (Informal):
Topic Model or Topic Twaddle? Re-evaluating Semantic Interpretability Measures (Doogan & Buntine, NAACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.naacl-main.300.pdf
Optional supplementary data:
 2021.naacl-main.300.OptionalSupplementaryData.zip
Video:
 https://aclanthology.org/2021.naacl-main.300.mp4