With the recent commercial availability of Large Language Models (LLMs), end users increasingly turn to them for encyclopedic knowledge. Doing so, users may inadvertently inject incorrect assumptions (as presuppositions) into LLM prompts, which are then affirmed by LLMs. In this work, we investigate the effect that existential presuppositions have on LLM responses. We introduce the Dataset for Evaluating and Explaining the Effect of Existential Presuppositions (D4EP) and use it to quantify their effect. We further develop a probing-based explanation that analyzes the LLM as a classifier and finds that the effect of presuppositions corresponds to a change of the classifier’s operating point.

Solidarity is a crucial concept to understand social relations in societies. In this paper, we explore fine-grained solidarity frames to study solidarity towards women and migrants in German parliamentary debates between 1867 and 2022. Using 2,864 manually annotated text snippets (with a cost exceeding 18k Euro), we evaluate large language models (LLMs) like Llama 3, GPT-3.5, and GPT-4. We find that GPT-4 outperforms other LLMs, approaching human annotation quality. Using GPT-4, we automatically annotate more than 18k further instances (with a cost of around 500 Euro) across 155 years and find that solidarity with migrants outweighs anti-solidarity but that frequencies and solidarity types shift over time. Most importantly, group-based notions of (anti-)solidarity fade in favor of compassionate solidarity, focusing on the vulnerability of migrant groups, and exchange-based anti-solidarity, focusing on the lack of (economic) contribution. Our study highlights the interplay of historical events, socio-economic needs, and political ideologies in shaping migration discourse and social cohesion. We also show that powerful LLMs, if carefully prompted, can be cost-effective alternatives to human annotation for hard social scientific tasks.