HinglishEval Generation Challenge on Quality Estimation of Synthetic Code-Mixed Text: Overview and Results

Vivek Srivastava, Mayank Singh


Abstract
We hosted a shared task to investigate the factors influencing the quality of the code- mixed text generation systems. The teams experimented with two systems that gener- ate synthetic code-mixed Hinglish sentences. They also experimented with human ratings that evaluate the generation quality of the two systems. The first-of-its-kind, proposed sub- tasks, (i) quality rating prediction and (ii) an- notators’ disagreement prediction of the syn- thetic Hinglish dataset made the shared task quite popular among the multilingual research community. A total of 46 participants com- prising 23 teams from 18 institutions reg- istered for this shared task. The detailed description of the task and the leaderboard is available at https://codalab.lisn.upsaclay.fr/competitions/1688.
Anthology ID:
2022.inlg-genchal.3
Volume:
Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges
Month:
July
Year:
2022
Address:
Waterville, Maine, USA and virtual meeting
Editors:
Samira Shaikh, Thiago Ferreira, Amanda Stent
Venue:
INLG
SIG:
SIGGEN
Publisher:
Association for Computational Linguistics
Note:
Pages:
19–25
Language:
URL:
https://aclanthology.org/2022.inlg-genchal.3
DOI:
Bibkey:
Cite (ACL):
Vivek Srivastava and Mayank Singh. 2022. HinglishEval Generation Challenge on Quality Estimation of Synthetic Code-Mixed Text: Overview and Results. In Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges, pages 19–25, Waterville, Maine, USA and virtual meeting. Association for Computational Linguistics.
Cite (Informal):
HinglishEval Generation Challenge on Quality Estimation of Synthetic Code-Mixed Text: Overview and Results (Srivastava & Singh, INLG 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.inlg-genchal.3.pdf