MIPE: A Metric Independent Pipeline for Effective Code-Mixed NLG Evaluation

Ayush Garg, Sammed Kagi, Vivek Srivastava, Mayank Singh


Abstract
Code-mixing is a phenomenon of mixing words and phrases from two or more languages in a single utterance of speech and text. Due to the high linguistic diversity, code-mixing presents several challenges in evaluating standard natural language generation (NLG) tasks. Various widely popular metrics perform poorly with the code-mixed NLG tasks. To address this challenge, we present a metric in- dependent evaluation pipeline MIPE that significantly improves the correlation between evaluation metrics and human judgments on the generated code-mixed text. As a use case, we demonstrate the performance of MIPE on the machine-generated Hinglish (code-mixing of Hindi and English languages) sentences from the HinGE corpus. We can extend the proposed evaluation strategy to other code-mixed language pairs, NLG tasks, and evaluation metrics with minimal to no effort.
Anthology ID:
2021.eval4nlp-1.13
Volume:
Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Editors:
Yang Gao, Steffen Eger, Wei Zhao, Piyawat Lertvittayakumjorn, Marina Fomicheva
Venue:
Eval4NLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
123–132
Language:
URL:
https://aclanthology.org/2021.eval4nlp-1.13
DOI:
10.18653/v1/2021.eval4nlp-1.13
Bibkey:
Cite (ACL):
Ayush Garg, Sammed Kagi, Vivek Srivastava, and Mayank Singh. 2021. MIPE: A Metric Independent Pipeline for Effective Code-Mixed NLG Evaluation. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 123–132, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
MIPE: A Metric Independent Pipeline for Effective Code-Mixed NLG Evaluation (Garg et al., Eval4NLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.eval4nlp-1.13.pdf
Video:
 https://aclanthology.org/2021.eval4nlp-1.13.mp4