Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learning

Lucas Weber, Elia Bruni, Dieuwke Hupkes


Abstract
Finding the best way of adapting pre-trained language models to a task is a big challenge in current NLP. Just like the previous generation of task-tuned models (TT), models that are adapted to tasks via in-context-learning (ICL) or instruction tuning (IT) are robust in some setups, but not in others. Here, we present a detailed analysis of which design choices cause instabilities and inconsistencies in LLM predictions. First, we show how spurious correlations between input distributions and labels – a known issue in TT models – form only a minor problem for prompted models. Then we engage in a systematic, holistic evaluation of different factors that have been found to influence predictions in a prompting setup. We test all possible combinations of a range of factors on both vanilla and instruction-tuned LLMs of different scale, and statistically analyse the results to show which factors are the most influential, the most interactive or the most stable. From our results, we deduce which factors can be used without precautions, should be avoided or handled with care in most settings.
Anthology ID:
2023.conll-1.20
Volume:
Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL)
Month:
December
Year:
2023
Address:
Singapore
Editors:
Jing Jiang, David Reitter, Shumin Deng
Venue:
CoNLL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
294–313
Language:
URL:
https://aclanthology.org/2023.conll-1.20
DOI:
10.18653/v1/2023.conll-1.20
Bibkey:
Cite (ACL):
Lucas Weber, Elia Bruni, and Dieuwke Hupkes. 2023. Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learning. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 294–313, Singapore. Association for Computational Linguistics.
Cite (Informal):
Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learning (Weber et al., CoNLL 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.conll-1.20.pdf