Multilingual Medical Reasoning for Question Answering with Large Language Models

Pietro Ferrazzi, Aitor Soroa, Rodrigo Agerri


Abstract
Large Language Models with reasoning capabilities have recently shown strong promise for medical Question Answering. Current approaches report competitive results on English benchmarks through knowledge distillation from large models, yet multilingual coverage remains limited. We extend this line of research by presenting a method to generate multilingual reasoning traces that enrich the distillation paradigm with factual information from Wikipedia. Using a retrieval-augmented generation pipeline over WikiProject Medicine, we produce 500k reasoning traces in English, Italian, and Spanish. The traces are generated to solve medical questions drawn from MedQA and MedMCQA, both of which we extend to Italian and Spanish. We evaluate our approach across in-domain and out-of-domain Medical QA benchmarks, demonstrating consistent performance gains in both few-shot in-context learning and supervised fine-tuning settings, achieving state-of-the-art results among 8B-parameter models across multiple model families. With the aim of fostering the development of multilingual clinical decision-support tools, we release the complete suite of resources: reasoning traces, translated datasets, Medical Wikipedia, and fine-tuned models.
Anthology ID:
2026.findings-emnlp.1105
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2026
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
23365–23387
Language:
URL:
https://aclanthology.org/2026.findings-emnlp.1105/
DOI:
Bibkey:
Cite (ACL):
Pietro Ferrazzi, Aitor Soroa, and Rodrigo Agerri. 2026. Multilingual Medical Reasoning for Question Answering with Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 23365–23387, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Multilingual Medical Reasoning for Question Answering with Large Language Models (Ferrazzi et al., Findings 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.findings-emnlp.1105.pdf