Adapting large language models to individual users remains challenging because systems must capture fine-grained personal preferences while serving many users efficiently. We introduce CARD, a hierarchical personalization framework that first learns preferences shared across user groups and then captures how individual users depart from those shared patterns. CARD trains a parameter-efficient adaptation module for each group, providing a reliable prior even when a user has limited history. It then learns a compact user preference vector by contrasting user-authored responses with outputs from the corresponding group model, without requiring manual annotations. During inference, both the backbone and the group adaptation module remain frozen, while the user vector applies a low-rank correction to the token logits during decoding. Experiments on personalization benchmarks show that CARD consistently outperforms competitive baselines while requiring only a compact vector for each user.