Tamer Soliman
2024
Correcting Language Model Outputs by Editing Salient Layers
Kshitij Mishra
|
Tamer Soliman
|
Anil Ramakrishna
|
Aram Galstyan
|
Anoop Kumar
Findings of the Association for Computational Linguistics: EACL 2024
Large language models can accumulate incorrect or outdated knowledge as the real world evolves. Compared to typical solutions such as retraining, retrieval augmented generation, model editing offers an effective yet low cost solution to address this issue. However, existing model editing algorithms employ manual selection of edit layers, which requires prior domain knowledge or expensive architecture-specific empirical layer selection methods, such as causal tracing. In this work, we propose SaLEM (Salient Layers Editing Model), an efficient solution for data driven layer selection for the model editing task. Our solution utilizes layer-wise saliency maps for layer selection, and matches the accuracy of prior approaches but with only 1/3 of their edits, enabling efficient updates to the parametric knowledge in large language models.
Leveraging LLMs for Dialogue Quality Measurement
Jinghan Jia
|
Abi Komma
|
Timothy Leffel
|
Xujun Peng
|
Ajay Nagesh
|
Tamer Soliman
|
Aram Galstyan
|
Anoop Kumar
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track)
In task-oriented conversational AI evaluation, unsupervised methods poorly correlate with human judgments, and supervised approaches lack generalization. Recent advances in large language models (LLMs) show robust zero- and few-shot capabilities across NLP tasks. Our paper explores using LLMs for automated dialogue quality evaluation, experimenting with various configurations on public and proprietary datasets. Manipulating factors such as model size, in-context examples, and selection techniques, we examine “chain-of-thought” (CoT) reasoning and label extraction procedures. Our results show that (1) larger models yield more accurate dialogue labels; (2) algorithmic selection of in-context examples outperforms random selection,; (3) CoT reasoning where an LLM is asked to provide justifications before outputting final labels improves performance; and (4) fine-tuned LLMs outperform out-of-the-box ones. In addition, we find that suitably tuned LLMs exhibit high accuracy in dialogue evaluation compared to human judgments.
Search
Co-authors
- Aram Galstyan 2
- Anoop Kumar 2
- Kshitij Mishra 1
- Anil Ramakrishna 1
- Jinghan Jia 1
- show all...