ALBETO and DistilBETO: Lightweight Spanish Language Models

José Cañete, Sebastian Donoso, Felipe Bravo-Marquez, Andrés Carvallo, Vladimir Araujo


Abstract
In recent years there have been considerable advances in pre-trained language models, where non-English language versions have also been made available. Due to their increasing use, many lightweight versions of these models (with reduced parameters) have also been released to speed up training and inference times. However, versions of these lighter models (e.g., ALBERT, DistilBERT) for languages other than English are still scarce. In this paper we present ALBETO and DistilBETO, which are versions of ALBERT and DistilBERT pre-trained exclusively on Spanish corpora. We train several versions of ALBETO ranging from 5M to 223M parameters and one of DistilBETO with 67M parameters. We evaluate our models in the GLUES benchmark that includes various natural language understanding tasks in Spanish. The results show that our lightweight models achieve competitive results to those of BETO (Spanish-BERT) despite having fewer parameters. More specifically, our larger ALBETO model outperforms all other models on the MLDoc, PAWS-X, XNLI, MLQA, SQAC and XQuAD datasets. However, BETO remains unbeaten for POS and NER. As a further contribution, all models are publicly available to the community for future research.
Anthology ID:
2022.lrec-1.457
Volume:
Proceedings of the Thirteenth Language Resources and Evaluation Conference
Month:
June
Year:
2022
Address:
Marseille, France
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
4291–4298
Language:
URL:
https://aclanthology.org/2022.lrec-1.457
DOI:
Bibkey:
Cite (ACL):
José Cañete, Sebastian Donoso, Felipe Bravo-Marquez, Andrés Carvallo, and Vladimir Araujo. 2022. ALBETO and DistilBETO: Lightweight Spanish Language Models. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4291–4298, Marseille, France. European Language Resources Association.
Cite (Informal):
ALBETO and DistilBETO: Lightweight Spanish Language Models (Cañete et al., LREC 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.lrec-1.457.pdf
Code
 OpenCENIA/lightweight-spanish-language-models
Data
CoNLL 2002GLUEMLDocMLQAPAWS-XSQuAD-esUniversal DependenciesXNLIXQuAD