BinarySelect to Improve Accessibility of Black-Box Attack Research

Shatarupa Ghosh, Jonathan Rusert


Abstract
Adversarial text attack research is useful for testing the robustness of NLP models, however, the rise of transformers has greatly increased the time required to test attacks. Especially when researchers do not have access to adequate resources (e.g. GPUs). This can hinder attack research, as modifying one example for an attack can require hundreds of queries to a model, especially for black-box attacks. Often these attacks remove one token at a time to find the ideal one to change, requiring n queries (the length of the text) right away. We propose a more efficient selection method called BinarySelect which combines binary search and attack selection methods to greatly reduce the number of queries needed to find a token. We find that BinarySelect only needs log_2(n) * 2 queries to find the first token compared to n queries. We also test BinarySelect in an attack setting against 5 classifiers across 3 datasets and find a viable tradeoff between number of queries saved and attack effectiveness. For example, on the Yelp dataset, the number of queries is reduced by 32% (72 less) with a drop in attack effectiveness of only 5 points. We believe that BinarySelect can help future researchers study adversarial attacks and black-box problems more efficiently and opens the door for researchers with access to less resources.
Anthology ID:
2025.coling-main.728
Volume:
Proceedings of the 31st International Conference on Computational Linguistics
Month:
January
Year:
2025
Address:
Abu Dhabi, UAE
Editors:
Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, Steven Schockaert
Venue:
COLING
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
10960–10976
Language:
URL:
https://aclanthology.org/2025.coling-main.728/
DOI:
Bibkey:
Cite (ACL):
Shatarupa Ghosh and Jonathan Rusert. 2025. BinarySelect to Improve Accessibility of Black-Box Attack Research. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10960–10976, Abu Dhabi, UAE. Association for Computational Linguistics.
Cite (Informal):
BinarySelect to Improve Accessibility of Black-Box Attack Research (Ghosh & Rusert, COLING 2025)
Copy Citation:
PDF:
https://aclanthology.org/2025.coling-main.728.pdf