Whether intermediate reasoning is computationally useful or merely explanatory depends on whether chain-of-thought (CoT) tokens contain task-solving information. We present a mechanistic causal analysis of CoT on GSM8K using _activation patching_: transferring token-level hidden states from an arithmetic CoT generation to a direct-answer run for the same question, then measuring the effect on final-answer accuracy. Across models, patching can elicit generations that lead to correct answers, revealing that individual CoT hidden states can encode sufficient task-solving information even when the original CoT trace is incorrect. This information is more prevalent in correct than incorrect CoT runs and is unevenly distributed across tokens, concentrating in mid-to-late layers and appearing earlier in the reasoning trace. Moreover, patching from language tokens such as verbs and entities more frequently elicits correct reasoning traces, whereas patches from mathematical tokens more often produce answer-proximal generations that rarely succeed. Further analysis shows that earlier CoT tokens concentrate recoverable information, while later ones require broader search to find the correct answer. Together, these findings show that CoT encodes recoverable, token-level task-solving information, revealing how arithmetic reasoning is represented, accessed, and disrupted.

Evaluating Large Language Models (LLMs) on reasoning benchmarks demonstrates their ability to solve compositional questions. However, little is known of whether these models engage in genuine logical reasoning or simply rely on implicit cues to generate answers. In this paper, we investigate the transitive reasoning capabilities of two distinct LLM architectures, LLaMA 2 and Flan-T5, by manipulating facts within two compositional datasets: QASC and Bamboogle. We controlled for potential cues that might influence the models’ performance, including (a) word/phrase overlaps across sections of test input; (b) models’ inherent knowledge during pre-training or fine-tuning; and (c) Named Entities. Our findings reveal that while both models leverage (a), Flan-T5 shows more resilience to experiments (b and c), having less variance than LLaMA 2. This suggests that models may develop an understanding of transitivity through fine-tuning on knowingly relevant datasets, a hypothesis we leave to future work.

Several studies have investigated the reasons behind the effectiveness of fine-tuning, usually through the lens of probing. However, these studies often neglect the role of the size of the dataset on which the model is fine-tuned. In this paper, we highlight the importance of this factor and its undeniable role in probing performance. We show that the extent of encoded linguistic knowledge depends on the number of fine-tuning samples. The analysis also reveals that larger training data mainly affects higher layers, and that the extent of this change is a factor of the number of iterations updating the model during fine-tuning rather than the diversity of the training samples. Finally, we show through a set of experiments that fine-tuning data size affects the recoverability of the changes made to the model’s linguistic knowledge.