Large Reasoning Models (LRMs) have demonstrated exceptional capabilities in complex problem-solving, largely driven by emergent reflection and backtracking mechanisms. However, understanding these mechanisms through mechanistic interpretability remains challenging. Current sparsity-based approaches rely heavily on heuristic keywords (e.g., "wait") and focus on isolated features, lacking a holistic view of how these features collaborate. In this paper, we propose a novel interpretability framework that bridges the microscopic and macroscopic analysis of Sparse Auto-Encoder (SAE) features. By leveraging an auxiliary LRM to robustly annotate reasoning trajectories, we introduce a comprehensive "Reflection Score" to quantify a feature’s relevance to reflection without keyword bias. Our geometric analysis of these features reveals that LRMs encode reflection not as discrete, isolated vectors, but within continuous regions of the hidden state space. Furthermore, we uncover an intrinsic, difficulty-aware reflection representation mechanism: the model dynamically activates sub-regions based on problem complexity. Building upon these insights, we introduce a static, difficulty-informed steering strategy that suppresses superficial, simple-problem oriented reflection features during inference. Comprehensive evaluations across three LRMs and challenging reasoning benchmarks demonstrate that our intervention simultaneously improves reasoning accuracy by an average of 2.25% and reduces token consumption by 5.3%. Our findings provide a deeper geometric understanding of LRM reflection and offer an interpretable, scalable approach to optimizing LRM reasoning.

This paper introduces a pioneering methodology, termed StructTuning, to efficiently transform foundation Large Language Models (LLMs) into domain specialists. It significantly reduces the training corpus needs to a mere 5% while achieving an impressive 100% of traditional knowledge injection performance. Motivated by structured human education, we propose a novel two-stage strategy for knowledge injection and alignment: Structure-aware Continual Pre-Training (SCPT) and Structure-aware Supervised Fine-Tuning (SSFT). In the SCPT phase, we automatically extract the domain knowledge taxonomy and reorganize the training corpora, enabling LLMs to effectively link textual segments to targeted knowledge points within the taxonomy. In the SSFT phase, we explicitly prompt models to elucidate the underlying knowledge structure in their outputs, leveraging the structured domain insight to address practical problems. Our ultimate method was extensively evaluated across model architectures and scales on LongBench and MMedBench datasets, demonstrating superior performance against other knowledge injection methods. We also explored our method’s scalability across different training corpus sizes, laying the foundation to enhance domain-specific LLMs with better data utilization.

Recent advancements in large language models (LLMs) have shown promising ability to perform commonsense reasoning, bringing machines closer to human-like understanding. However, deciphering the internal reasoning processes of LLMs remains challenging due to the complex interdependencies among generated tokens, especially in practical question-answering. In this study, we introduce a two-dimensional analysis framework—comprising token back-tracing and individual token decoding—to uncover how LLMs conduct factual knowledge recall. Through explanatory analysis of three typical reasoning datasets, we identify a consistent three-phase pattern: Subject Augmentation and Broadcasting, Object Retrieval and Reranking, and Conclusion Fusion and Generation. Our findings reveal that LLMs do not lack relevant knowledge but struggle to select the most accurate information based on context during the retrieval and rerank phase. Leveraging these findings, we apply representation engineering and selective fine-tuning to target specific modules responsible for retrieval and rerank errors. Experimental results show large improvements in response accuracy for both in-domain and out-of-domain settings, validating the rationality of the interpreting result.