Community OSCAR: A Community Effort for Multilingual Web Data

Manuel Brack, Malte Ostendorff, Pedro Ortiz Suarez, José Javier Saiz, Iñaki Lacunza Castilla, Jorge Palomar-Giner, Alexander Shvets, Patrick Schramowski, Georg Rehm, Marta Villegas, Kristian Kersting


Abstract
The development of large language models (LLMs) relies heavily on extensive, high-quality datasets. Publicly available datasets focus predominantly on English, leaving other language communities behind. To address this issue, we introduce Community OSCAR, a multilingual dataset initiative designed to address the gap between English and non-English data availability. Through a collective effort, Community OSCAR covers over 150 languages with 45 billion documents, totaling over 345 TiB of data. Initial results indicate that Community OSCAR provides valuable raw data for training LLMs and enhancing the performance of multilingual models. This work aims to contribute to the ongoing advancements in multilingual NLP and to support a more inclusive AI ecosystem by making high-quality, multilingual data more accessible to those working with low-resource languages.
Anthology ID:
2024.mrl-1.19
Volume:
Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024)
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Jonne Sälevä, Abraham Owodunni
Venue:
MRL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
232–235
Language:
URL:
https://aclanthology.org/2024.mrl-1.19
DOI:
10.18653/v1/2024.mrl-1.19
Bibkey:
Cite (ACL):
Manuel Brack, Malte Ostendorff, Pedro Ortiz Suarez, José Javier Saiz, Iñaki Lacunza Castilla, Jorge Palomar-Giner, Alexander Shvets, Patrick Schramowski, Georg Rehm, Marta Villegas, and Kristian Kersting. 2024. Community OSCAR: A Community Effort for Multilingual Web Data. In Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024), pages 232–235, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
Community OSCAR: A Community Effort for Multilingual Web Data (Brack et al., MRL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.mrl-1.19.pdf