CDLM: Cross-Document Language Modeling

Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew Peters, Arie Cattan, Ido Dagan


Abstract
We introduce a new pretraining approach geared for multi-document language modeling, incorporating two key ideas into the masked language modeling self-supervised objective. First, instead of considering documents in isolation, we pretrain over sets of multiple related documents, encouraging the model to learn cross-document relationships. Second, we improve over recent long-range transformers by introducing dynamic global attention that has access to the entire input to predict masked tokens. We release CDLM (Cross-Document Language Model), a new general language model for multi-document setting that can be easily applied to downstream tasks. Our extensive analysis shows that both ideas are essential for the success of CDLM, and work in synergy to set new state-of-the-art results for several multi-text tasks.
Anthology ID:
2021.findings-emnlp.225
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2021
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
2648–2662
Language:
URL:
https://aclanthology.org/2021.findings-emnlp.225
DOI:
10.18653/v1/2021.findings-emnlp.225
Bibkey:
Cite (ACL):
Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew Peters, Arie Cattan, and Ido Dagan. 2021. CDLM: Cross-Document Language Modeling. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2648–2662, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
CDLM: Cross-Document Language Modeling (Caciularu et al., Findings 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.findings-emnlp.225.pdf
Video:
 https://aclanthology.org/2021.findings-emnlp.225.mp4
Code
 aviclu/cdlm +  additional community code
Data
ECB+Multi-NewsOCS2ORC