In the time series domain, increasing efforts extend large language models (LLMs) to multimodal large language models (MLLMs) by incorporating temporal data for downstream temporal understanding tasks. In conventional MLLM training, models primarily rely on language signals, while temporal information provides substantially weaker supervision. This imbalance leads to insufficient learning of temporal patterns. To address this issue, we propose TimeSense, a framework with a disentangled reconstruction structure that introduces explicit temporal supervision and encourages the learning of both global trends and local temporal variations that are often underrepresented in text-dominated multimodal training. Existing QA benchmarks provide limited coverage of temporal variations and compositional temporal dependencies, making it difficult to systematically diagnose these MLLMs capabilities across diverse temporal understanding tasks. To facilitate a more fine-grained evaluation, we introduce EvalTS, a comprehensive benchmark spanning three difficulty levels (Atomic, Molecular, and Compositional) with diverse high-quality queries for evaluating temporal understanding abilities in large language and multimodal language models. Experiments on EvalTS and existing benchmarks demonstrate that TimeSense consistently outperforms prior methods and substantially improves temporal understanding performance.