LLM agents increasingly operate in long-horizon tasks and extended dialogues, where they must retrieve useful past experiences beyond limited context windows. A central challenge is to balance efficient recall with faithful evidence recovery. Existing memory methods often rely on static access strategies: compact memories reduce context cost but may omit fine-grained evidence, whereas full histories preserve details but introduce redundant and noisy context. We propose GraP-Mem, a multi-granularity memory planning framework that adaptively schedules memory access for each question. GraP-Mem represents each historical session with two complementary views: compact semantic memories for efficient retrieval and source contexts for detailed evidence recovery. During inference, a Plan Agent generates retrieval-oriented information needs and queries, while an Integration Agent maintains a question-specific evidence state that consolidates useful evidence and identifies remaining gaps. When the evidence is incomplete, GraP-Mem selectively expands access to source contexts or performs additional retrieval. The final Answer Model is conditioned only on the integrated evidence state. Experiments on LoCoMo and NarrativeQA show that GraP-Mem consistently improves F1 and BLEU across backbone models, demonstrating the effectiveness of plan-guided memory access across multiple granularities.