Investigating Mysteries of CoT-Augmented Distillation

Somin Wadhwa, Silvio Amir, Byron C Wallace


Abstract
Eliciting chain of thought (CoT) rationales - sequences of token that convey a “reasoning” process has been shown to consistently improve LLM performance on tasks like question answering. More recent efforts have shown that such rationales can also be used for model distillation: Including CoT sequences (elicited from a large “teacher” model) in addition to target labels when fine-tuning a small student model yields (often substantial) improvements. In this work we ask: Why and how does this additional training signal help in model distillation? We perform ablations to interrogate this, and report some potentially surprising results. Specifically: (1) Placing CoT sequences after labels (rather than before) realizes consistently better downstream performance – this means that no student “reasoning” is necessary at test time to realize gains. (2) When rationales are appended in this way, they need not be coherent reasoning sequences to yield improvements; performance increases are robust to permutations of CoT tokens, for example. In fact, (3) a small number of key tokens are sufficient to achieve improvements equivalent to those observed when full rationales are used in model distillation.
Anthology ID:
2024.emnlp-main.349
Volume:
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6071–6086
Language:
URL:
https://aclanthology.org/2024.emnlp-main.349
DOI:
10.18653/v1/2024.emnlp-main.349
Bibkey:
Cite (ACL):
Somin Wadhwa, Silvio Amir, and Byron C Wallace. 2024. Investigating Mysteries of CoT-Augmented Distillation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6071–6086, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
Investigating Mysteries of CoT-Augmented Distillation (Wadhwa et al., EMNLP 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.emnlp-main.349.pdf