Tool Quality, Not Framework Complexity: Evaluating RLM Vision, Code, and Memory Pipelines for Radiology Applications

Jonas Schewior, Ansari Hamid Raja Gulam, Mohamed Ashfaq Anwerdeen, Andreas Wendemuth


Abstract
It remains unclear whether agentic augmentation (iterative tool use, code execution, or retrieved memory) improves over single-pass VLM inference for medical image interpretation. We present a systematic quantitative evaluation of the Recursive Language Model (RLM) framework on a multimodal medical task, comparing three pipeline architectures (P1 direct vision, P2 iterative RLM tool use, P3 model-written code agent) across four retrieval-augmented memory conditions on 100 CheXpert Plus chest X-rays, scoring with CheXbert (14-label classification) and GREEN (radiology-specific LLM judge). Three findings emerge: (1) direct vision outperforms both agentic variants; the RLM loop consumes 7-9× more tokens without adding task-relevant information when its only tool calls the same VLM; (2) retrieval consistently helps, but the optimal pool size depends on metric and pipeline: top-5 is the sweet spot for direct-vision report quality (GREEN +37%), while 24 retrieved cases further benefit pathology classification (CheXbert +50%) and can improve GREEN for agentic pipelines; and (3) qualitative analysis shows code agents with a pretrained classifier succeed when it returns confident pathology scores but fail on normal cases where it returns noise-floor outputs, illustrating that tool quality, not framework complexity, determines outcome.
Anthology ID:
2026.realm-1.26
Volume:
Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Ehsan Kamalloo, Nicolas Gontier, Nikolai Rozanov, Maryam Hashemzadeh, Akhil Arora
Venues:
REALM | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
382–397
Language:
URL:
https://aclanthology.org/2026.realm-1.26/
DOI:
Bibkey:
Cite (ACL):
Jonas Schewior, Ansari Hamid Raja Gulam, Mohamed Ashfaq Anwerdeen, and Andreas Wendemuth. 2026. Tool Quality, Not Framework Complexity: Evaluating RLM Vision, Code, and Memory Pipelines for Radiology Applications. In Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026), pages 382–397, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Tool Quality, Not Framework Complexity: Evaluating RLM Vision, Code, and Memory Pipelines for Radiology Applications (Schewior et al., REALM 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.realm-1.26.pdf