The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail

Samuel Bowman


Abstract
Researchers in NLP often frame and discuss research results in ways that serve to deemphasize the field’s successes, often in response to the field’s widespread hype. Though well-meaning, this has yielded many misleading or false claims about the limits of our best technology. This is a problem, and it may be more serious than it looks: It harms our credibility in ways that can make it harder to mitigate present-day harms, like those involving biased systems for content moderation or resume screening. It also limits our ability to prepare for the potentially enormous impacts of more distant future advances. This paper urges researchers to be careful about these claims and suggests some research directions and communication strategies that will make it easier to avoid or rebut them.
Anthology ID:
2022.acl-long.516
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7484–7499
Language:
URL:
https://aclanthology.org/2022.acl-long.516
DOI:
10.18653/v1/2022.acl-long.516
Bibkey:
Cite (ACL):
Samuel Bowman. 2022. The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7484–7499, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail (Bowman, ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-long.516.pdf
Video:
 https://aclanthology.org/2022.acl-long.516.mp4
Data
SQuAD