Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability

Afra Feyza Akyürek, Ekin Akyürek, Leshem Choshen, Derry Wijaya, Jacob Andreas


Abstract
While language models (LMs) can sometimes generate factually correct text and estimate truth values of individual claims, these generally do not reflect a globally coherent, manipulable model of the world. As a consequence, current LMs also generate incorrect or nonsensical content, and are difficult to edit and bring up to date. We present a method called Deductive Closure Training (DCT) that uses LMs themselves to identify implications of (and contradictions within) the text that they generate, yielding an efficient self-supervised procedure for improving LM factuality. Given a collection of seed documents, DCT prompts LMs to generate additional text implied by these documents, reason globally about the correctness of this generated text, and finally fine-tune on text inferred to be correct. Given seed documents from a trusted source, DCT provides a tool for supervised model updating; if seed documents are sampled from the LM itself, DCT enables fully unsupervised fine-tuning for improved coherence and accuracy. Across the CREAK, MQuAKE, and Reversal Curse datasets, supervised DCT improves LM fact verification and text generation accuracy by 3-26%; on CREAK, fully unsupervised DCT improves verification accuracy by 12%. These results show that LMs’ reasoning capabilities during inference can be leveraged during training to improve their reliability.
Anthology ID:
2024.findings-acl.584
Volume:
Findings of the Association for Computational Linguistics: ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
9802–9818
Language:
URL:
https://aclanthology.org/2024.findings-acl.584
DOI:
10.18653/v1/2024.findings-acl.584
Bibkey:
Cite (ACL):
Afra Feyza Akyürek, Ekin Akyürek, Leshem Choshen, Derry Wijaya, and Jacob Andreas. 2024. Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9802–9818, Bangkok, Thailand. Association for Computational Linguistics.
Cite (Informal):
Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability (Akyürek et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-acl.584.pdf