Can GPT-4 Recover Latent Semantic Relational Information from Word Associations? A Detailed Analysis of Agreement with Human-annotated Semantic Ontologies.

Simon De Deyne, Chunhua Liu, Lea Frermann


Abstract
Word associations, i.e., spontaneous responses to a cue word, provide not only a window into the human mental lexicon but have also been shown to be a repository of common-sense knowledge and can underpin efforts in lexicography and the construction of dictionaries. Especially the latter tasks require knowledge about the relations underlying the associations (e.g., Taxonomic vs. Situational); however, to date, there is neither an established ontology of relations nor an effective labelling paradigm. Here, we test GPT-4’s ability to infer semantic relations for human-produced word associations. We use four human-labelled data sets of word associations and semantic features, with differing relation inventories and various levels of annotator agreement. We directly prompt GPT-4 with detailed relation definitions without further fine-tuning or training. Our results show that while GPT-4 provided a good account of higher-level classifications (e.g. Taxonomic vs Situational), prompting instructions alone cannot obtain similar performance for detailed classifications (e.g. superordinate, subordinate or coordinate relations) despite high agreement among human annotators. This suggests that latent relations can at least be partially recovered from word associations and highlights ways in which LLMs could be improved and human annotation protocols could adapted to reduce coding ambiguity.
Anthology ID:
2024.cogalex-1.8
Volume:
Proceedings of the Workshop on Cognitive Aspects of the Lexicon @ LREC-COLING 2024
Month:
May
Year:
2024
Address:
Torino, Italia
Editors:
Michael Zock, Emmanuele Chersoni, Yu-Yin Hsu, Simon de Deyne
Venue:
CogALex
SIG:
Publisher:
ELRA and ICCL
Note:
Pages:
68–78
Language:
URL:
https://aclanthology.org/2024.cogalex-1.8
DOI:
Bibkey:
Cite (ACL):
Simon De Deyne, Chunhua Liu, and Lea Frermann. 2024. Can GPT-4 Recover Latent Semantic Relational Information from Word Associations? A Detailed Analysis of Agreement with Human-annotated Semantic Ontologies.. In Proceedings of the Workshop on Cognitive Aspects of the Lexicon @ LREC-COLING 2024, pages 68–78, Torino, Italia. ELRA and ICCL.
Cite (Informal):
Can GPT-4 Recover Latent Semantic Relational Information from Word Associations? A Detailed Analysis of Agreement with Human-annotated Semantic Ontologies. (De Deyne et al., CogALex 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.cogalex-1.8.pdf