Rina Zviel-Girshin
2023
Using C-LARA to evaluate GPT-4’s multilingual processing
ChatGPT C-LARA-Instance
|
Belinda Chiera
|
Cathy Chua
|
Chadi Raheb
|
Manny Rayner
|
Annika Simonsen
|
Zhengkang Xiang
|
Rina Zviel-Girshin
Proceedings of the 21st Annual Workshop of the Australasian Language Technology Association
We present a cross-linguistic study in which the open source C-LARA platform was used to evaluate GPT-4’s ability to perform several key tasks relevant to Computer Assisted Language Learning. For each of the languages English, Farsi, Faroese, Mandarin and Russian, we instructed GPT-4, through C-LARA, to write six different texts, using prompts chosen to obtain texts of widely differing character. We then further instructed GPT-4 to annotate each text with segmentation markup, glosses and lemma/part-of-speech information; native speakers hand-corrected the texts and annotations to obtain error rates on the different component tasks. The C-LARA platform makes it easy to combine the results into a single multimodal document, further facilitating checking of their correctness. GPT-4’s performance varied widely across languages and processing tasks, but performance on different text genres was roughly comparable. In some cases, most notably glossing of English text, we found that GPT-4 was consistently able to revise its annotations to improve them.
Search
Co-authors
- ChatGPT C-LARA-Instance 1
- Belinda Chiera 1
- Cathy Chua 1
- Chadi Raheb 1
- Manny Rayner 1
- show all...
Venues
- alta1