The consultation of reference cases in clinical radiology reporting motivates integrating multimodal retrieval-augmented generation (RAG) into automated systems. However, traditional RAG approaches employ single-vector embeddings for indexing, resulting in loss of granular information. Moreover, a semantic gap exists whereby visually similar radiographs often encode distinct pathologies. In this paper, we propose a Semantics-guided Multi-modal Late Interaction RAG (SeMLI-RAG) framework for radiology report generation. Our approach employs a weight-based late interaction strategy for retrieval to capture fine-grained radiographic features. Furthermore, we introduce an image-adaptive RAG to selectively leverage retrieved information, mitigating performance degradation caused by retrieval-induced noise. Experiments on three radiology benchmarks demonstrate that our framework yields consistent improvements over the backbone and RAG baselines across multiple evaluation dimensions. Further analysis shows that improving retrieval granularity and retrieval robustness can both benefit radiology report generation.