Stop Word Lists in Free Open-source Software Packages

Joel Nothman, Hanmin Qin, Roman Yurchak


Abstract
Open-source software packages for language processing often include stop word lists. Users may apply them without awareness of their surprising omissions (e.g. “hasn’t” but not “hadn’t”) and inclusions (“computer”), or their incompatibility with a particular tokenizer. Motivated by issues raised about the Scikit-learn stop list, we investigate variation among and consistency within 52 popular English-language stop lists, and propose strategies for mitigating these issues.
Anthology ID:
W18-2502
Volume:
Proceedings of Workshop for NLP Open Source Software (NLP-OSS)
Month:
July
Year:
2018
Address:
Melbourne, Australia
Editors:
Eunjeong L. Park, Masato Hagiwara, Dmitrijs Milajevs, Liling Tan
Venue:
NLPOSS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7–12
Language:
URL:
https://aclanthology.org/W18-2502
DOI:
10.18653/v1/W18-2502
Bibkey:
Cite (ACL):
Joel Nothman, Hanmin Qin, and Roman Yurchak. 2018. Stop Word Lists in Free Open-source Software Packages. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 7–12, Melbourne, Australia. Association for Computational Linguistics.
Cite (Informal):
Stop Word Lists in Free Open-source Software Packages (Nothman et al., NLPOSS 2018)
Copy Citation:
PDF:
https://aclanthology.org/W18-2502.pdf
Software:
 W18-2502.Software.zip