Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondence

Myeongjun Jang, Frank Mtumbuka, Thomas Lukasiewicz


Abstract
The logical negation property (LNP), which implies generating different predictions for semantically opposite inputs (p is true iff ¬p is false), is an important property that a trustworthy language model must satisfy. However, much recent evidence shows that large-size pre-trained language models (PLMs) do not satisfy this property. In this paper, we perform experiments using probing tasks to assess PLMs’ LNP understanding. Unlike previous studies that only examined negation expressions, we expand the boundary of the investigation to lexical semantics. Through experiments, we observe that PLMs violate the LNP frequently. To alleviate the issue, we propose a novel intermediate training task, named meaning-matching, designed to directly learn a meaning text correspondence, instead of relying on the distributional hypothesis. Through multiple experiments, we find that the task enables PLMs to learn lexical semantic information. Also, through fine-tuning experiments on 7 GLUE tasks, we confirm that it is a safe intermediate task that guarantees a similar or better performance of downstream tasks. Finally, we observe that our proposed approach outperforms our previous counterparts despite its time and resource efficiency.
Anthology ID:
2022.findings-naacl.156
Volume:
Findings of the Association for Computational Linguistics: NAACL 2022
Month:
July
Year:
2022
Address:
Seattle, United States
Venues:
Findings | NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2030–2042
Language:
URL:
https://aclanthology.org/2022.findings-naacl.156
DOI:
10.18653/v1/2022.findings-naacl.156
Bibkey:
Cite (ACL):
Myeongjun Jang, Frank Mtumbuka, and Thomas Lukasiewicz. 2022. Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondence. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2030–2042, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondence (Jang et al., Findings 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.findings-naacl.156.pdf
Code
 mj-jang/beyond-distributional
Data
ConceptNetGLUELAMAQNLISNLISuperGLUE