Personalizing large language models (LLMs) requires reflecting stable user preferences while adapting their expression to each target context. Soft-prompt personalization has become a practical route to this goal, but existing methods produce soft prompts in which stable preferences, target-induced content, and context-dependent expression are bundled into an entangled conditioning signal without separately addressable components, leaving inference with no principled space to intervene on individual aspects of personalization. We propose Factorized soft prompts with Latent Control (FaLCon), an LLM personalization framework built on two coupled stages. First, factorized soft prompts replace this entangled conditioning with a set of separately addressable preference factors, forming a structured control interface. Second, inference-time latent control treats the resulting factor weights as a continuous action space and adapts them per instance from teacher-forced likelihood feedback, keeping the LLM frozen and updating only a low-dimensional factor-weight posterior without back-propagating through the LLM. Experiments on multiple personalized generation tasks reveal that FaLCon consistently outperforms strong baselines, validating the individual necessity and joint effectiveness of the structured interface and the inference-time controller.