Long-term conversational agents often face time-sensitive queries, requiring evidence that is both contextually relevant and valid for a specific time, event, or factual update. However, noisy temporal-entity cues, weak coupling between temporal anchors and retrieval paths, and sparse long-tail evidence often lead memory systems to retrieve semantically similar but temporally invalid or outdated evidence. We propose TSHMem, a Temporal-Semantic Hierarchical Memory Graph for long-term conversational agents. TSHMem addresses noisy conversational evidence by organizing dialogue turns, self-contained statements, hierarchical summaries, and entities into a heterogeneous memory graph. To connect temporal constraints with retrieval paths, it parses temporal intent, activates query-relevant entities, resolves event-referenced constraints, and injects temporal relevance into graph transition probabilities. It then combines time-aware structural propagation with semantic matching and hierarchical ranking to retrieve compact evidence that is temporally valid, contextually relevant, and robust to sparse long-tail memories. Experiments on LongMemEval S* and LoCoMo show that TSHMem consistently outperforms strong retrieval and memory-agent baselines, achieving 66.3% accuracy on LongMemEval S*, with an absolute gain of 18.6% over the strongest baseline while retrieving fewer evidence tokens.