Rina Zviel-Girshin


2023

pdf bib
Using C-LARA to evaluate GPT-4’s multilingual processing
ChatGPT C-LARA-Instance | Belinda Chiera | Cathy Chua | Chadi Raheb | Manny Rayner | Annika Simonsen | Zhengkang Xiang | Rina Zviel-Girshin
Proceedings of the 21st Annual Workshop of the Australasian Language Technology Association

We present a cross-linguistic study in which the open source C-LARA platform was used to evaluate GPT-4’s ability to perform several key tasks relevant to Computer Assisted Language Learning. For each of the languages English, Farsi, Faroese, Mandarin and Russian, we instructed GPT-4, through C-LARA, to write six different texts, using prompts chosen to obtain texts of widely differing character. We then further instructed GPT-4 to annotate each text with segmentation markup, glosses and lemma/part-of-speech information; native speakers hand-corrected the texts and annotations to obtain error rates on the different component tasks. The C-LARA platform makes it easy to combine the results into a single multimodal document, further facilitating checking of their correctness. GPT-4’s performance varied widely across languages and processing tasks, but performance on different text genres was roughly comparable. In some cases, most notably glossing of English text, we found that GPT-4 was consistently able to revise its annotations to improve them.