What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus

Alexandra Luccioni, Joseph Viviano


Abstract
Whereas much of the success of the current generation of neural language models has been driven by increasingly large training corpora, relatively little research has been dedicated to analyzing these massive sources of textual data. In this exploratory analysis, we delve deeper into the Common Crawl, a colossal web corpus that is extensively used for training language models. We find that it contains a significant amount of undesirable content, including hate speech and sexually explicit content, even after filtering procedures. We discuss the potential impacts of this content on language models and conclude with future research directions and a more mindful approach to corpus collection and analysis.
Anthology ID:
2021.acl-short.24
Volume:
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)
Month:
August
Year:
2021
Address:
Online
Venues:
ACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
182–189
Language:
URL:
https://aclanthology.org/2021.acl-short.24
DOI:
10.18653/v1/2021.acl-short.24
Bibkey:
Cite (ACL):
Alexandra Luccioni and Joseph Viviano. 2021. What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 182–189, Online. Association for Computational Linguistics.
Cite (Informal):
What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus (Luccioni & Viviano, ACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.acl-short.24.pdf
Video:
 https://aclanthology.org/2021.acl-short.24.mp4