Xiaocheng Luo

Author directory

2026

Unlike human reasoning in abstract conceptual spaces, large language models (LLMs) typically reason by generating discrete tokens at each step, then discard the rich semantic information contained in the output distribution. Recent work such as Soft Thinking has shown that LLMs’ latent reasoning via soft concepts is a promising direction, but LLMs are trained on discrete tokens, leaving the continuous concept space underutilized during training. To address this, we propose Soft Concept Mixing (SCM), a soft concept aware training scheme that directly exposes the model to soft representations during training. Specifically, SCM constructs a soft concept vector by forming a probability-weighted average of embeddings. Then, this vector is mixed into the model’s hidden states, which embody rich contextual information. Finally, the entire latent reasoning process is optimized with Reinforcement Learning (RL). Experiments on six reasoning benchmarks demonstrate that SCM improves the reasoning performance of LLMs, and simultaneously maintains a stable training dynamic.