MFAQ: a Multilingual FAQ Dataset

Maxime De Bruyn, Ehsan Lotfi, Jeska Buhmann, Walter Daelemans


Abstract
In this paper, we present the first multilingual FAQ dataset publicly available. We collected around 6M FAQ pairs from the web, in 21 different languages. Although this is significantly larger than existing FAQ retrieval datasets, it comes with its own challenges: duplication of content and uneven distribution of topics. We adopt a similar setup as Dense Passage Retrieval (DPR) and test various bi-encoders on this dataset. Our experiments reveal that a multilingual model based on XLM-RoBERTa achieves the best results, except for English. Lower resources languages seem to learn from one another as a multilingual model achieves a higher MRR than language-specific ones. Our qualitative analysis reveals the brittleness of the model on simple word changes. We publicly release our dataset, model, and training script.
Anthology ID:
2021.mrqa-1.1
Volume:
Proceedings of the 3rd Workshop on Machine Reading for Question Answering
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Venues:
EMNLP | MRQA
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1–13
Language:
URL:
https://aclanthology.org/2021.mrqa-1.1
DOI:
10.18653/v1/2021.mrqa-1.1
Bibkey:
Cite (ACL):
Maxime De Bruyn, Ehsan Lotfi, Jeska Buhmann, and Walter Daelemans. 2021. MFAQ: a Multilingual FAQ Dataset. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 1–13, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
MFAQ: a Multilingual FAQ Dataset (De Bruyn et al., MRQA 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.mrqa-1.1.pdf
Code
 clips/mfaq
Data
MFAQPAQ