Reinforcement learning has become a common approach for improving language-model reasoning, but its effectiveness depends heavily on reward design, especially for compact domain-specialized models. We study structured biomedical question answering with two 4B-scale models, MedGemma-1.5-4B-it and Qwen3-4B-Thinking-2507, using a two-stage SFT+GRPO pipeline that trains models to produce tagged think/answer outputs. We compare three reward schemes: a simple format + accuracy reward, a depth incentivized reward that encourages multiple reasoning cycles, and a gated LLM-judge reward for intermediate reasoning milestones. Across five biomedical QA benchmarks, the simplest reward is the most robust: it substantially improves MedGemma in the structured-output setting, while depth and LLM-judge rewards often reduce accuracy. For Qwen3, SFT or native long-budget inference can outperform GRPO on several benchmarks, showing that reasoning-tuned models may not benefit from additional structural reinforcement. Our results suggest that, for lightweight biomedical QA, reward simplicity and model priors are more important than increasingly elaborate intermediate reasoning supervision. The codebase for this study is publicly available on GitHub: https://github.com/batuinmetz/When-more-supervision-hurts.