Viviane Binet
2025
Q&A-LF : A French Question-Answering Benchmark for Measuring Fine-Grained Lexical Knowledge
Alexander Petrov
|
Alessandra Thais Mancas
|
Viviane Binet
|
Antoine Venant
|
Francois Lareau
|
Yves Lepage
|
Phillippe Langlais
Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing - Natural Language Processing in the Generative AI Era
We introduce Q&A-LF, a French, question-answering benchmark designed to assess the extent to which large language models capture fine-grained lexical knowledge. We investigate the ability of ChatGPT-4o mini, Qwen2.5-14B, Llama3.0-8B, and Llama3.1-8B to answer questions based on lexical functions from Meaning-Text Theory. Using various prompting setups with different levels of examples and context, we find that Qwen and ChatGPT generally outperform Llama models, achieving up to 70% accuracy, while Llama models reach just above 60%. We identify LFs that are particularly easy or especially challenging for the models. We further investigate whether providing sentence-level context and one-shot prompting improve performance, especially on semantically complex functions.
Search
Fix author
Co-authors
- Philippe Langlais 1
- François Lareau 1
- Yves Lepage 1
- Alessandra Thais Mancas 1
- Alexander Petrov 1
- show all...