Exploratory Model Analysis Using Data-Driven Neuron Representations

Daisuke Oba, Naoki Yoshinaga, Masashi Toyoda


Abstract
Probing classifiers have been extensively used to inspect whether a model component captures specific linguistic phenomena. This top-down approach is, however, costly when we have no probable hypothesis on the association between the target model component and phenomena. In this study, aiming to provide a flexible, exploratory analysis of a neural model at various levels ranging from individual neurons to the model as a whole, we present a bottom-up approach to inspect the target neural model by using neuron representations obtained from a massive corpus of text. We first feed massive amount of text to the target model and collect sentences that strongly activate each neuron. We then abstract the collected sentences to obtain neuron representations that help us interpret the corresponding neurons; we augment the sentences with linguistic annotations (e.g., part-of-speech tags) and various metadata (e.g., topic and sentiment), and apply pattern mining and clustering techniques to the augmented sentences. We demonstrate the utility of our method by inspecting the pre-trained BERT. Our exploratory analysis reveals that i) specific phrases and domains of text are captured by individual neurons in BERT, ii) a group of neurons simultaneously capture the same linguistic phenomena, and iii) deeper-level layers capture more specific linguistic phenomena.
Anthology ID:
2021.blackboxnlp-1.41
Original:
2021.blackboxnlp-1.41v1
Version 2:
2021.blackboxnlp-1.41v2
Volume:
Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Editors:
Jasmijn Bastings, Yonatan Belinkov, Emmanuel Dupoux, Mario Giulianelli, Dieuwke Hupkes, Yuval Pinter, Hassan Sajjad
Venue:
BlackboxNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
518–528
Language:
URL:
https://aclanthology.org/2021.blackboxnlp-1.41
DOI:
10.18653/v1/2021.blackboxnlp-1.41
Bibkey:
Cite (ACL):
Daisuke Oba, Naoki Yoshinaga, and Masashi Toyoda. 2021. Exploratory Model Analysis Using Data-Driven Neuron Representations. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 518–528, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Exploratory Model Analysis Using Data-Driven Neuron Representations (Oba et al., BlackboxNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.blackboxnlp-1.41.pdf
Data
BookCorpus