Abhineet Rajendra Kulkarni

Author directory

2026

We investigate how small open-weight language models make decisions in medical tasks and whether their failures also occur in other settings. We evaluated nine models on wearable fatigue and intensive-care tasks, where they chose among six recommendations, including continued monitoring. We then extended testing to a non-medical tool router and subsets of the Berkeley Function-Calling Leaderboard and JSONSchemaBench. We introduce the Minimum Viable Audit (MVA) to examine which actions models choose, whether their responses contain usable decisions, and whether they copy prompt content. Comparisons across tasks help establish where these behaviors recur. Under the baseline intensive-care prompt, MedGemma chose continued monitoring in all 450 attempts, achieving 33.3% accuracy. On the router, it used all six tools and answered 144 of 150 requests correctly. Meditron-7B produced unusable responses and copied prompt content across intensive-care monitoring and the non-medical evaluations at both four-bit and eight-bit quantization. MedGemma’s results motivate checking the medical instructions. Meditron’s recurring failures warrant checking the model configuration and how outputs are generated. Differences between tasks and mismatches between medical prompts and scoring rules limit conclusions about the causes of these failures. We recommend reporting selected actions, invalid and copied responses, and results for each prompt condition alongside accuracy.