Multilingual Open Text Release 1: Public Domain News in 44 Languages

Chester Palen-Michel, June Kim, Constantine Lignos


Abstract
We present a Multilingual Open Text (MOT), a new multilingual corpus containing text in 44 languages, many of which have limited existing text resources for natural language processing. The first release of the corpus contains over 2.8 million news articles and an additional 1 million short snippets (photo captions, video descriptions, etc.) published between 2001–2022 and collected from Voice of America’s news websites. We describe our process for collecting, filtering, and processing the data. The source material is in the public domain, our collection is licensed using a creative commons license (CC BY 4.0), and all software used to create the corpus is released under the MIT License. The corpus will be regularly updated as additional documents are published.
Anthology ID:
2022.lrec-1.224
Volume:
Proceedings of the Thirteenth Language Resources and Evaluation Conference
Month:
June
Year:
2022
Address:
Marseille, France
Editors:
Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
2080–2089
Language:
URL:
https://aclanthology.org/2022.lrec-1.224
DOI:
Bibkey:
Cite (ACL):
Chester Palen-Michel, June Kim, and Constantine Lignos. 2022. Multilingual Open Text Release 1: Public Domain News in 44 Languages. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2080–2089, Marseille, France. European Language Resources Association.
Cite (Informal):
Multilingual Open Text Release 1: Public Domain News in 44 Languages (Palen-Michel et al., LREC 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.lrec-1.224.pdf
Code
 VietHoang1710/khmer-nltk +  additional community code
Data
OSCAR