LMdiff: A Visual Diff Tool to Compare Language Models

Hendrik Strobelt, Benjamin Hoover, Arvind Satyanaryan, Sebastian Gehrmann


Abstract
While different language models are ubiquitous in NLP, it is hard to contrast their outputs and identify which contexts one can handle better than the other. To address this question, we introduce LMdiff, a tool that visually compares probability distributions of two models that differ, e.g., through finetuning, distillation, or simply training with different parameter sizes. LMdiff allows the generation of hypotheses about model behavior by investigating text instances token by token and further assists in choosing these interesting text instances by identifying the most interesting phrases from large corpora. We showcase the applicability of LMdiff for hypothesis generation across multiple case studies. A demo is available at http://lmdiff.net .
Anthology ID:
2021.emnlp-demo.12
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
96–105
Language:
URL:
https://aclanthology.org/2021.emnlp-demo.12
DOI:
10.18653/v1/2021.emnlp-demo.12
Bibkey:
Cite (ACL):
Hendrik Strobelt, Benjamin Hoover, Arvind Satyanaryan, and Sebastian Gehrmann. 2021. LMdiff: A Visual Diff Tool to Compare Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 96–105, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
LMdiff: A Visual Diff Tool to Compare Language Models (Strobelt et al., EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-demo.12.pdf
Video:
 https://aclanthology.org/2021.emnlp-demo.12.mp4
Code
 hendrikstrobelt/lmdiff
Data
CommonsenseQAWinoBias