Ignacio Chaparro
Author directory2026
Assessing the Comprehension of Regional Colombian Idioms by Language Models
Santiago Bobadilla Suarez | Juan Diego Osorio | Maria Alejandra Pinzon Roncancio | Ignacio Chaparro | Ruben Manrique
Proceedings of the 3rd Workshop on Language Understanding in the Human-Machine Era (LUHME 2026)
Santiago Bobadilla Suarez | Juan Diego Osorio | Maria Alejandra Pinzon Roncancio | Ignacio Chaparro | Ruben Manrique
Proceedings of the 3rd Workshop on Language Understanding in the Human-Machine Era (LUHME 2026)
This study queries 23 large language models (LLMs) and reports analyses for 22 models on Colombian idioms and regionally marked expressions. We introduce an evaluation set derived from Colombian lexicographical sources and assess models on three tasks: positive-class idiom recognition, definition generation, and context-based paraphrase generation. Automatic metrics show substantial variation across models, with Gemini-3-Flash-Preview achieving the highest average score on the meaning-oriented tasks. Human evaluation of four selected models qualifies these rankings: the best Task 2 definition accuracy reached only 58.57%, while Task 3 accuracy was higher for all four models. Exploratory regional analysis of the tagged subset shows uneven performance across Colombian macro-regions, with lower results for expressions from the Llanos and high Caribbean regions. These findings suggest that current LLMs only partially capture Colombian idiomatic meaning and motivate more regionally balanced resources and evaluation protocols for Spanish.