Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistently

Kanishka Misra, Allyson Ettinger, Kyle Mahowald


Abstract
Recent zero-shot evaluations have highlighted important limitations in the abilities of language models (LMs) to perform meaning extraction. However, it is now well known that LMs can demonstrate radical improvements in the presence of experimental contexts such as in-context examples and instructions. How well does this translate to previously studied meaning-sensitive tasks? We present a case-study on the extent to which experimental contexts can improve LMs’ robustness in performing property inheritance—predicting semantic properties of novel concepts, a task that they have been previously shown to fail on. Upon carefully controlling the nature of the in-context examples and the instructions, our work reveals that they can indeed lead to non-trivial property inheritance behavior in LMs. However, this ability is inconsistent: with a minimal reformulation of the task, some LMs were found to pick up on shallow, non-semantic heuristics from their inputs, suggesting that the computational principles of semantic property inference are yet to be mastered by LMs.
Anthology ID:
2024.emnlp-main.687
Volume:
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
12342–12355
Language:
URL:
https://aclanthology.org/2024.emnlp-main.687
DOI:
10.18653/v1/2024.emnlp-main.687
Bibkey:
Cite (ACL):
Kanishka Misra, Allyson Ettinger, and Kyle Mahowald. 2024. Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistently. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12342–12355, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistently (Misra et al., EMNLP 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.emnlp-main.687.pdf
Software:
 2024.emnlp-main.687.software.zip