How Does the Experimental Setting Affect the Conclusions of Neural Encoding Models?

Xiaohan Zhang, Shaonan Wang, Chengqing Zong


Abstract
Recent years have witnessed the tendency of neural encoding models on exploring brain language processing using naturalistic stimuli. Neural encoding models are data-driven methods that require an encoding model to investigate the mystery of brain mechanisms hidden in the data. As a data-driven method, the performance of encoding models is very sensitive to the experimental setting. However, it is unknown how the experimental setting further affects the conclusions of neural encoding models. This paper systematically investigated this problem and evaluated the influence of three experimental settings, i.e., the data size, the cross-validation training method, and the statistical testing method. Results demonstrate that inappropriate cross-validation training and small data size can substantially decrease the performance of encoding models, especially in the temporal lobe and the frontal lobe. And different null hypotheses in significance testing lead to highly different significant brain regions. Based on these results, we suggest a block-wise cross-validation training method and an adequate data size for increasing the performance of linear encoding models. We also propose two strict null hypotheses to control false positive discovery rates.
Anthology ID:
2022.lrec-1.687
Volume:
Proceedings of the Thirteenth Language Resources and Evaluation Conference
Month:
June
Year:
2022
Address:
Marseille, France
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
6397–6404
Language:
URL:
https://aclanthology.org/2022.lrec-1.687
DOI:
Bibkey:
Cite (ACL):
Xiaohan Zhang, Shaonan Wang, and Chengqing Zong. 2022. How Does the Experimental Setting Affect the Conclusions of Neural Encoding Models?. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6397–6404, Marseille, France. European Language Resources Association.
Cite (Informal):
How Does the Experimental Setting Affect the Conclusions of Neural Encoding Models? (Zhang et al., LREC 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.lrec-1.687.pdf