Referring Remote Sensing Image Segmentation (RRSIS) aims to segment the corresponding target region in a remote sensing image according to a natural-language expression. Compared with conventional referring image segmentation, RRSIS faces more significant scale variations, complex spatial layouts, and visual distractions, placing higher demands on vision–language alignment and multi-scale modeling. To address these challenges, we propose DLSA, a dual-level semantic alignment framework for RRSIS. DLSA introduces semantic alignment at two complementary levels, namely encoding and decoding. At the encoder level, we design Semantic-guided Adaptive Multi-scale Convolution (SgAMC), which explicitly introduces linguistic semantics into receptive-field selection, enabling language-guided adaptive scale modeling. At the decoder level, we introduce Adaptive Late-stage Text Injection (ALTI), which adaptively injects textual semantics into late decoding stages to refine mask reconstruction while preserving the stability of the decoding pathway. Experiments on RRSIS-D and RefSegRS demonstrate that DLSA achieves strong performance, validating the effectiveness of dual-level semantic alignment in remote sensing image scenarios. The source code is publicly available at https://github.com/young-qkl/DLSA.