Decoding throughput gains from larger inference batches are often limited by GPU memory, mainly due to the large size of KV cache. Training-free KV cache offloading mitigates this issue by storing redundant context on the CPU and fetching sparse KV subsets, but often harms long-generation quality due to training-inference mismatch of sparse patterns. Trainable sparse attention can improve quality, yet its unconstrained KV accesses are inefficient for offloading and may incur excessive CPU–GPU transfers. To this end, we propose NOSA, a trainable sparse attention mechanism designed for KV cache offloading. NOSA explicitly bounds CPU–GPU KV transfers, enabling low communication overhead and high decoding throughput. We further develop NOSI, an offloading inference system that realizes these efficiency gains. Experiments on 1,3,8B LLMs show that NOSA outperforms KV cache offloading baselines on general, long-input, and long-generation tasks, while improving decoding throughput by up to 5.04×, 1.92×, and 1.83× over FullAttn, InfLLMv2, and ShadowKV, respectively.