Roger Bautista
2024
An Evaluation of English to Spanish Medical Translation by Large Language Models
Nicholas Riina
|
Likhitha Patlolla
|
Camilo Hernandez Joya
|
Roger Bautista
|
Melissa Olivar-Villanueva
|
Anish Kumar
Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 2: Presentations)
Machine translation (MT) with Large Language Models (LLMs) holds promise as a clinical translation tool with more capabilities than a traditional MT model. This work compares the quality of English to Spanish translation by three LLMs: ChatGPT3.5 Turbo, ChatGPT4o, and Aguila, against Google Translate. The test set used in this study is MedlinePlus, a parallel dataset of educational health information in English and Spanish developed by the National Library of Medicine. ChatGPT4o and Google Translate performed similarly in both automated scoring (BLEU, METEOR, and BERTscore) and human evaluation with ChatGPT3.5 Turbo not far behind. Aguila, the only LLM intended for primarily Spanish and Catalan use, surprisingly performed much worse than the other models. However, qualitative analysis of Aguila’s results revealed the use of Spanish word choice that may reach a broader audience.