The “political opinions” of Large Language Models (LLMs) increasingly find their way into political discourse despite limited under- standing and control of these systems. Our paper adds to the understanding of how prompt manipulation impacts the multilingual generation of political arguments, focusing on two fundamental aspects of generated text: style and semantics. Starting from a reference set of Swiss politicians’ arguments in German, Italian, and French, we conduct a large-scale prompt-manipulation experiment on 4 LLMs, building the prompts in a modular way (i.e., with blocks encoding socio-demographics, context, and examples) and testing all possible combinations. Using our parallel dataset of over 375k human argument-aligned generated arguments, we assess the detectability of LLM arguments through classification experiments with 160 linguistic features and semantic embeddings. We then carry out quantitative and qualitative analysis to understand how human- written vs. LLM-generated text compares, and to assess the impact of prompt manipulation.

Our results show that human and LLM arguments are substantially different in style and semantics, making their identification easy for classifiers. Prompt manipulation does not substantially impact semantics and style, except for sociodemographic persona prompting, which makes generated texts moderately more human-like. Model, language, and topics impact the stability of the generated text most. We make the code, data, and raw results available via https://github.com/mmmaurer/promptology.

Detecting hyperpartisan narratives and Population Replacement Conspiracy Theories (PRCT) is essential to addressing the spread of misinformation. These complex narratives pose a significant threat, as hyperpartisanship drives political polarisation and institutional distrust, while PRCTs directly motivate real-world extremist violence, making their identification critical for social cohesion and public safety. However, existing resources are scarce, predominantly English-centric, and often analyse hyperpartisanship, stance, and rhetorical bias in isolation rather than as interrelated aspects of political discourse. To bridge this gap, we introduce PartisanLens, the first multilingual dataset of 1617 hyperpartisan news headlines in Spanish, Italian, and Portuguese, annotated in multiple political discourse aspects. We first evaluate the classification performance of widely used Large Language Models (LLMs) on this dataset, establishing robust baselines for the classification of hyperpartisan and PRCT narratives. In addition, we assess the viability of using LLMs as automatic annotators for this task, analysing their ability to approximate human annotation. Results highlight both their potential and current limitations. Next, moving beyond standard judgments, we explore whether LLMs can emulate human annotation patterns by conditioning them on socio-economic and ideological profiles that simulate annotator perspectives. At last, we provide our resources and evaluation; PartisanLens supports future research on detecting partisan and conspiratorial narratives in European contexts.