The growing prevalence of multi-sensor wearable devices generates vast amounts of long-term, multimodal time-series data, posing significant challenges for context-aware Large Language Models (LLMs). Current LLM-based health analysis methods typically rely on manually curated context, which becomes impractical with increasing data volume and sensor diversity. To overcome these limitations, we introduce Wearable As Graph (WAG), a novel framework that automates context retrieval for LLMs using personalized knowledge graphs. WAG constructs knowledge graphs mapping relationships between wearable modalities and incorporates user-specific data. We develop a data-driven retrieval pipeline that leverages both global (long-term) and local (short-term) relationships within metrics to identify the most relevant nodes for user queries. We evaluate WAG on a benchmark of over 10k data-associated queries created from multiple wearable datasets. WAG consistently delivers the strongest overall performance, while using substantially fewer input tokens than query-window full-context prompting in the diagnostic setting. These gains remain broadly consistent under human evaluation and across a cross-model generator–judge matrix, supporting structured, query-adaptive context retrieval for LLM-driven analysis of wearable data.