Discovering influential text using convolutional neural networks

Megan Ayers, Luke Sanford, Margaret Roberts, Eddie Yang


Abstract
Experimental methods for estimating the impacts of text on human evaluation have been widely used in the social sciences. However, researchers in experimental settings are usually limited to testing a small number of pre-specified text treatments. While efforts to mine unstructured texts for features that causally affect outcomes have been ongoing in recent years, these models have primarily focused on the topics or specific words of text, which may not always be the mechanism of the effect. We connect these efforts with NLP interpretability techniques and present a method for flexibly discovering clusters of similar text phrases that are predictive of human reactions to texts using convolutional neural networks. When used in an experimental setting, this method can identify text treatments and their effects under certain assumptions. We apply the method to two data sets. The first enables direct validation of the model’s ability to detect phrases known to cause the outcome. The second demonstrates its ability to flexibly discover text treatments with varying textual structures. In both cases, the model learns a greater variety of text treatments compared to benchmark methods, and these text features quantitatively meet or exceed the ability of benchmark methods to predict the outcome.
Anthology ID:
2024.findings-acl.714
Volume:
Findings of the Association for Computational Linguistics: ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
12002–12027
Language:
URL:
https://aclanthology.org/2024.findings-acl.714
DOI:
10.18653/v1/2024.findings-acl.714
Bibkey:
Cite (ACL):
Megan Ayers, Luke Sanford, Margaret Roberts, and Eddie Yang. 2024. Discovering influential text using convolutional neural networks. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12002–12027, Bangkok, Thailand. Association for Computational Linguistics.
Cite (Informal):
Discovering influential text using convolutional neural networks (Ayers et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-acl.714.pdf