Learning Sentence Embeddings for Coherence Modelling and Beyond

Tanner Bohn, Yining Hu, Jinhang Zhang, Charles Ling


Abstract
We present a novel and effective technique for performing text coherence tasks while facilitating deeper insights into the data. Despite obtaining ever-increasing task performance, modern deep-learning approaches to NLP tasks often only provide users with the final network decision and no additional understanding of the data. In this work, we show that a new type of sentence embedding learned through self-supervision can be applied effectively to text coherence tasks while serving as a window through which deeper understanding of the data can be obtained. To produce these sentence embeddings, we train a recurrent neural network to take individual sentences and predict their location in a document in the form of a distribution over locations. We demonstrate that these embeddings, combined with simple visual heuristics, can be used to achieve performance competitive with state-of-the-art on multiple text coherence tasks, outperforming more complex and specialized approaches. Additionally, we demonstrate that these embeddings can provide insights useful to writers for improving writing quality and informing document structuring, and assisting readers in summarizing and locating information.
Anthology ID:
R19-1018
Volume:
Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)
Month:
September
Year:
2019
Address:
Varna, Bulgaria
Editors:
Ruslan Mitkov, Galia Angelova
Venue:
RANLP
SIG:
Publisher:
INCOMA Ltd.
Note:
Pages:
151–160
Language:
URL:
https://aclanthology.org/R19-1018
DOI:
10.26615/978-954-452-056-4_018
Bibkey:
Cite (ACL):
Tanner Bohn, Yining Hu, Jinhang Zhang, and Charles Ling. 2019. Learning Sentence Embeddings for Coherence Modelling and Beyond. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 151–160, Varna, Bulgaria. INCOMA Ltd..
Cite (Informal):
Learning Sentence Embeddings for Coherence Modelling and Beyond (Bohn et al., RANLP 2019)
Copy Citation:
PDF:
https://aclanthology.org/R19-1018.pdf