Don’t Search for a Search Method — Simple Heuristics Suffice for Adversarial Text Attacks

Nathaniel Berger, Stefan Riezler, Sebastian Ebert, Artem Sokolov


Abstract
Recently more attention has been given to adversarial attacks on neural networks for natural language processing (NLP). A central research topic has been the investigation of search algorithms and search constraints, accompanied by benchmark algorithms and tasks. We implement an algorithm inspired by zeroth order optimization-based attacks and compare with the benchmark results in the TextAttack framework. Surprisingly, we find that optimization-based methods do not yield any improvement in a constrained setup and slightly benefit from approximate gradient information only in unconstrained setups where search spaces are larger. In contrast, simple heuristics exploiting nearest neighbors without querying the target function yield substantial success rates in constrained setups, and nearly full success rate in unconstrained setups, at an order of magnitude fewer queries. We conclude from these results that current TextAttack benchmark tasks are too easy and constraints are too strict, preventing meaningful research on black-box adversarial text attacks.
Anthology ID:
2021.emnlp-main.647
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8216–8224
Language:
URL:
https://aclanthology.org/2021.emnlp-main.647
DOI:
10.18653/v1/2021.emnlp-main.647
Bibkey:
Cite (ACL):
Nathaniel Berger, Stefan Riezler, Sebastian Ebert, and Artem Sokolov. 2021. Don’t Search for a Search Method — Simple Heuristics Suffice for Adversarial Text Attacks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8216–8224, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Don’t Search for a Search Method — Simple Heuristics Suffice for Adversarial Text Attacks (Berger et al., EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.647.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.647.mp4