@inproceedings{guzman-etal-2019-flores,
    title = "The {FLORES} Evaluation Datasets for Low-Resource Machine Translation: {N}epali{--}{E}nglish and {S}inhala{--}{E}nglish",
    author = "Guzm{\'a}n, Francisco  and
      Chen, Peng-Jen  and
      Ott, Myle  and
      Pino, Juan  and
      Lample, Guillaume  and
      Koehn, Philipp  and
      Chaudhary, Vishrav  and
      Ranzato, Marc{'}Aurelio",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)",
    month = nov,
    year = "2019",
    address = "Hong Kong, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/D19-1632",
    doi = "10.18653/v1/D19-1632",
    pages = "6098--6111",
    abstract = "For machine translation, a vast majority of language pairs in the world are considered low-resource because they have little parallel data available. Besides the technical challenges of learning with limited supervision, it is difficult to evaluate methods trained on low-resource language pairs because of the lack of freely and publicly available benchmarks. In this work, we introduce the FLORES evaluation datasets for Nepali{--}English and Sinhala{--} English, based on sentences translated from Wikipedia. Compared to English, these are languages with very different morphology and syntax, for which little out-of-domain parallel data is available and for which relatively large amounts of monolingual data are freely available. We describe our process to collect and cross-check the quality of translations, and we report baseline performance using several learning settings: fully supervised, weakly supervised, semi-supervised, and fully unsupervised. Our experiments demonstrate that current state-of-the-art methods perform rather poorly on this benchmark, posing a challenge to the research community working on low-resource MT. Data and code to reproduce our experiments are available at https://github.com/facebookresearch/flores.",
}