The neural dynamics of word recognition and integration

Jon Gauthier, Roger Levy


Abstract
Listeners recognize and integrate words in rapid and noisy everyday speech by combining expectations about upcoming content with incremental sensory evidence. We present a computational model of word recognition which formalizes this perceptual process in Bayesian decision theory. We fit this model to explain scalp EEG signals recorded as subjects passively listened to a fictional story, revealing both the dynamics of the online auditory word recognition process and the neural correlates of the recognition and integration of words. The model reveals distinct neural processing of words depending on whether or not they can be quickly recognized. While all words trigger a neural response characteristic of probabilistic integration — voltage modulations predicted by a word’s surprisal in context — these modulations are amplified for words which require more than roughly 150 ms of input to be recognized. We observe no difference in the latency of these neural responses according to words’ recognition times. Our results support a two-part model of speech comprehension, combining an eager and rapid process of word recognition with a temporally independent process of word integration. However, we also developed alternative models of the scalp EEG signal not incorporating word recognition dynamics which showed similar performance improvements. We discuss potential future modeling steps which may help to separate these hypotheses.
Anthology ID:
2023.emnlp-main.62
Volume:
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
980–995
Language:
URL:
https://aclanthology.org/2023.emnlp-main.62
DOI:
10.18653/v1/2023.emnlp-main.62
Bibkey:
Cite (ACL):
Jon Gauthier and Roger Levy. 2023. The neural dynamics of word recognition and integration. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 980–995, Singapore. Association for Computational Linguistics.
Cite (Informal):
The neural dynamics of word recognition and integration (Gauthier & Levy, EMNLP 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.emnlp-main.62.pdf
Video:
 https://aclanthology.org/2023.emnlp-main.62.mp4