Radiology Report Generation (RRG) aims to reduce the burden of clinical documentation, yet generating reports that remain visually grounded in the input image remains challenging. Current medical vision-language models (Med-VLMs) often rely on curated preference data, retrieval augmentation, or large-scale labeled corpora, but remain prone to hallucinated findings due to weak cross-modal alignment. To address this challenge, we propose GRAIL: Grounded Radiology Alignment of Medical Vision-Language Models for Radiology Report Generation, a two-stage multimodal alignment framework for grounded RRG. First, Clinically Informed Textual Reasoning improves report structure and medical terminology using verifiable language rewards with periodic clinical guidance. Second, Expert-guided Visual Reasoning leverages a domain expert to rank generated reports using image-report alignment scores, encouraging visually grounded report generation without curated preference pairs or retrieval data. Experiments on multiple benchmarks show that GRAIL consistently improves both report generation quality and clinical accuracy, achieving relative gains over strong post-training baselines by up to 15% in clinical accuracy and 13% in generation quality. GRAIL also outperforms proprietary general-purpose and medical VLMs, highlighting the effectiveness of self-supervised multimodal post-alignment for reducing hallucinations in visually grounded radiology report generation.