Discourse-Structural Noise in LLM-Generated Explanations

Raghvi Baloni, Achim Rettinger


Abstract
Large language models (LLMs) are increasingly utilized to generate explanations of complex phenomena, yet existing evaluation frameworks assess quality extensively through semantic similarity to human references, a measure blind to discourse organization. We conduct a computational study of 300 ELI5 question pairs, comparing Llama-3.1 8B, Gemma-2 9B, and Mistral-7B against human-written responses using PDTB-grounded connective analysis across five relation types. All three models exhibit elaborative dominance, i.e. Elaborative connectives account for 56-66% of discourse markers versus 38% in human writing, with Causal, Contrastive, and Conditional relations systematically suppressed. This 18-28 percentage point gap persists after connective sense disambiguation (11-20 pp), ruling out measurement artifact. Across three semantic similarity metrics, PDTB connective noise explains less than 8% of shared variance (r2<0.08), confirming it captures evaluation signal invisible to existing approaches. We establish discourse-structural noise as a distinct, measurable evaluation dimension for LLM-generated explanations.
Anthology ID:
2026.wnut-1.5
Volume:
Proceedings of the 11th Workshop on Natural User-generated Text
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
JinYeong Bak, Rob van der Goot, Hyeju Jang, Weerayut Buaphet, Alan Ramponi, Wei Xu, Alan Ritter
Venues:
WNUT | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
65–71
Language:
URL:
https://aclanthology.org/2026.wnut-1.5/
DOI:
Bibkey:
Cite (ACL):
Raghvi Baloni and Achim Rettinger. 2026. Discourse-Structural Noise in LLM-Generated Explanations. In Proceedings of the 11th Workshop on Natural User-generated Text, pages 65–71, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Discourse-Structural Noise in LLM-Generated Explanations (Baloni & Rettinger, WNUT 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.wnut-1.5.pdf