Vincent Gripon

Author directory

2026

Discrete diffusion language models have emerged as a compelling alternative to autoregressive approaches, offering a range of structural advantages. This is due to their bidirectional attention allowing a global semantic view across intermediate denoising steps that enables flexible, order-agnostic generation. Despite this promise, they have received considerably less attention in ongoing research, and existing work has focused narrowly on mathematics and formal reasoning tasks. Their language comprehension capabilities, particularly for figurative language, remain entirely unevaluated, leaving open fundamental questions about how these models handle complex idiomatic structures, especially for languages other than English. We find that diffusion models do possess many strong idiomatic processing abilities, demonstrated through competitive token recovery in cloze tests and strong semantic ability from multiple-choice accuracy. However, this advantage fails to surface during inference as confidence-based demasking appears poorly suited for idioms, as early committed tokens override internal representations and discourage certain types of multi-token structures from generation, limiting their linguistic abilities.