Oliver Bott


2023

pdf bib
Beyond the Bias: Unveiling the Quality of Implicit Causality Prompt Continuations in Language Models
Judith Sieker | Oliver Bott | Torgrim Solstad | Sina Zarrieß
Proceedings of the 16th International Natural Language Generation Conference

Recent studies have used human continuations of Implicit Causality (IC) prompts collected in linguistic experiments to evaluate discourse understanding in large language models (LLMs), focusing on the well-known IC coreference bias in the LLMs’ predictions of the next word following the prompt. In this study, we investigate how continuations of IC prompts can be used to evaluate the text generation capabilities of LLMs in a linguistically controlled setting. We conduct an experiment using two open-source GPT-based models, employing human evaluation to assess different aspects of continuation quality. Our findings show that LLMs struggle in particular with generating coherent continuations in this rather simple setting, indicating a lack of discourse knowledge beyond the well-known IC bias. Our results also suggest that a bias congruent continuation does not necessarily equate to a higher continuation quality. Furthermore, our study draws upon insights from the Uniform Information Density hypothesis, testing different prompt modifications and decoding procedures and showing that sampling-based methods are particularly sensitive to the information density of the prompts.

2022

pdf bib
This isn’t the bias you’re looking for: Implicit causality, names and gender in German language models
Sina Zarrieß | Hannes Groener | Torgrim Solstad | Oliver Bott
Proceedings of the 18th Conference on Natural Language Processing (KONVENS 2022)

2011

pdf bib
Interpreting tractable versus intractable reciprocal sentences
Oliver Bott | Fabian Schlotterbeck | Jakub Szymanik
Proceedings of the Ninth International Conference on Computational Semantics (IWCS 2011)