CINO: A Chinese Minority Pre-trained Language Model

Ziqing Yang, Zihang Xu, Yiming Cui, Baoxin Wang, Min Lin, Dayong Wu, Zhigang Chen


Abstract
Multilingual pre-trained language models have shown impressive performance on cross-lingual tasks. It greatly facilitates the applications of natural language processing on low-resource languages. However, there are still some languages that the current multilingual models do not perform well on. In this paper, we propose CINO (Chinese Minority Pre-trained Language Model), a multilingual pre-trained language model for Chinese minority languages. It covers Standard Chinese, Yue Chinese, and six other ethnic minority languages. To evaluate the cross-lingual ability of the multilingual model on ethnic minority languages, we collect documents from Wikipedia and news websites, and construct two text classification datasets, WCM (Wiki-Chinese-Minority) and CMNews (Chinese-Minority-News). We show that CINO notably outperforms the baselines on various classification tasks. The CINO model and the datasets are publicly available at http://cino.hfl-rc.com.
Anthology ID:
2022.coling-1.346
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
3937–3949
Language:
URL:
https://aclanthology.org/2022.coling-1.346
DOI:
Bibkey:
Cite (ACL):
Ziqing Yang, Zihang Xu, Yiming Cui, Baoxin Wang, Min Lin, Dayong Wu, and Zhigang Chen. 2022. CINO: A Chinese Minority Pre-trained Language Model. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3937–3949, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
CINO: A Chinese Minority Pre-trained Language Model (Yang et al., COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.346.pdf
Data
CC100KLUE