Artsiom Saladukha
Author directory2026
MTGBench: A Multi-Turn Game Benchmark for Diverse LLM Capability Assessment
Alisa Alenicheva | Timur Zykov | Artsiom Saladukha | Vladislav Negodin
Findings of the Association for Computational Linguistics: EMNLP 2026
Alisa Alenicheva | Timur Zykov | Artsiom Saladukha | Vladislav Negodin
Findings of the Association for Computational Linguistics: EMNLP 2026
Existing LLM reasoning benchmarks are largely static, single-turn, English-centric, and reliant on biased LLM-as-a-judge scoring. We present MTGBench, a bilingual (English/Russian) suite of seven multi-turn puzzle games—Wordle, Word Chain, Situation Puzzles, 20 Questions, Wheel of Fortune, 20 Features, and Alias—spanning deductive, semantic, lateral, consistency, and knowledge-retrieval reasoning. Four games use algorithmic checkers; the remaining three use a constrained LLM-as-a-rule-enforcer validated against human annotators, and episodes are regenerated natively per language rather than translated. Across 16 frontier models and a human baseline comprising 476 sessions we find that (i) rankings are highly task-dependent, (ii) chain-of-thought is near-essential for letter-level games but uneven on knowledge-driven ones, and (iii) the strongest reasoning-enabled LLMs match or exceed our collected Russian human baseline across the entire suite. Game scores align most consistently with cognitive batteries probing Causal Reasoning, Pragmatics, Planning, and Common-sense Reasoning, though these associations are game-specific rather than a single executive-function signal. Code and data are available in MTGBench github.