Eye Gaze and Self-attention: How Humans and Transformers Attend Words in Sentences

Joshua Bensemann, Alex Peng, Diana Benavides-Prado, Yang Chen, Neset Tan, Paul Michael Corballis, Patricia Riddle, Michael Witbrock


Abstract
Attention describes cognitive processes that are important to many human phenomena including reading. The term is also used to describe the way in which transformer neural networks perform natural language processing. While attention appears to be very different under these two contexts, this paper presents an analysis of the correlations between transformer attention and overt human attention during reading tasks. An extensive analysis of human eye tracking datasets showed that the dwell times of human eye movements were strongly correlated with the attention patterns occurring in the early layers of pre-trained transformers such as BERT. Additionally, the strength of a correlation was not related to the number of parameters within a transformer. This suggests that something about the transformers’ architecture determined how closely the two measures were correlated.
Anthology ID:
2022.cmcl-1.9
Volume:
Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics
Month:
May
Year:
2022
Address:
Dublin, Ireland
Venue:
CMCL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
75–87
Language:
URL:
https://aclanthology.org/2022.cmcl-1.9
DOI:
10.18653/v1/2022.cmcl-1.9
Bibkey:
Cite (ACL):
Joshua Bensemann, Alex Peng, Diana Benavides-Prado, Yang Chen, Neset Tan, Paul Michael Corballis, Patricia Riddle, and Michael Witbrock. 2022. Eye Gaze and Self-attention: How Humans and Transformers Attend Words in Sentences. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 75–87, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Eye Gaze and Self-attention: How Humans and Transformers Attend Words in Sentences (Bensemann et al., CMCL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.cmcl-1.9.pdf
Video:
 https://aclanthology.org/2022.cmcl-1.9.mp4
Data
GLUEMovieQASuperGLUE