Mousa Arraf

Author directory

2026

Question answering (QA) systems should remain stable across meaning-preserving variations of the same problem. Earlier robustness work showed brittleness to local lexical perturbations, such as synonym replacement, word swaps, insertions, and deletions. Revisiting this robustness axis for LLMs, we find that high-performing models are now comparatively stable to such edits. However, this stability does not extend to discourse-level rewrites: answer-preserving transformations that alter a question’s structure and presentation while preserving its intent and gold answer. Across reasoning-focused QA benchmarks, these rewrites induce substantially larger performance drops than local perturbations. They also change how models fail: using a compact failure taxonomy and Total Variation distance, we show that discourse-level rewrites systematically redistribute error types, revealing distinct failure regimes. Our results suggest that local lexical perturbations remain useful, while discourse-level rewrites provide a complementary and stronger diagnostic signal for evaluating reasoning stability in QA LLMs.