Low-Cost Generation and Evaluation of Dictionary Example Sentences

Bill Cai, Ng Clarence, Daniel Liang, Shelvia Hotama


Abstract
Dictionary example sentences play an important role in illustrating word definitions and usage, but manually creating quality sentences is challenging. Prior works have demonstrated that language models can be trained to generate example sentences. However, they relied on costly customized models and word sense datasets for generation and evaluation of their work. Rapid advancements in foundational models present the opportunity to create low-cost, zero-shot methods for the generation and evaluation of dictionary example sentences. We introduce a new automatic evaluation metric called OxfordEval that measures the win-rate of generated sentences against existing Oxford Dictionary sentences. OxfordEval shows high alignment with human judgments, enabling large-scale automated quality evaluation. We experiment with various LLMs and configurations to generate dictionary sentences across word classes. We complement this with a novel approach of using masked language models to identify and select sentences that best exemplify word meaning. The eventual model, FM-MLM, achieves over 85.1% win rate against Oxford baseline sentences according to OxfordEval, compared to 39.8% win rate for prior model-generated sentences.
Anthology ID:
2024.naacl-long.194
Volume:
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3538–3549
Language:
URL:
https://aclanthology.org/2024.naacl-long.194
DOI:
Bibkey:
Cite (ACL):
Bill Cai, Ng Clarence, Daniel Liang, and Shelvia Hotama. 2024. Low-Cost Generation and Evaluation of Dictionary Example Sentences. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3538–3549, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Low-Cost Generation and Evaluation of Dictionary Example Sentences (Cai et al., NAACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.naacl-long.194.pdf
Copyright:
 2024.naacl-long.194.copyright.pdf