The LM Playschool Challenge 2026

Sherzod Hakimov, Dimitrije Ristic, Philipp Sadler, Davide Mazzaccara, Sabrina McCallum, Filippo Momentè, Maria Manina, Karl Osswald, Dieu-Thu Le, Oliver Lemon, Alexander Koller, Raffaella Bernardi, Raquel Fernández, Mario Giulianelli, Alessandro Suglia, David Schlangen

Abstract
Children acquire language from far less input than language models, and social interaction is widely thought to be part of the reason. Dialogue games cannot fully replicate human interaction, but they offer a controllable environment for simulating situated language use and for studying how models learn from and through interaction. The LM Playschool Challenge builds on this idea, shifting language model training away from observational data towards multi-turn language use. Participants were free to post-train a language model of any size, using any data or method, with one objective in mind — to improve its performance on an unseen set of dialogue games. Our findings suggest that post-training on dialogue games effectively teaches game rules and structure and can support generalisation, especially for small models — the best system achieved gains of roughly seven and three times its baseline score on held-out instances of seen games and on unseen games, respectively. However, gains on unseen games were concentrated in a few games, and larger models improved mainly by achieving better completion rates. General ability was largely retained across submitted systems, owing to the fact that fine-tuning on dialogue games produces shorter responses — and in turn, trigger fewer formatting errors on static benchmarks. Nearly all submissions relied on supervised fine-tuning, often followed by preference optimisation, and none trained on the dense turn-level feedback offered by Playpen, leaving this an open direction for future iterations of the challenge.
Anthology ID:
2026.lmplayschool-1.9
Volume:
The Proceedings of Improving Language Models through Learning from Dialogue Game Interaction
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Alessandro Suglia, Mario Giulianelli, David Schlangen, Raffaella Bernardi, Raquel Fernández, Sherzod Hakimov, Sabrina McCallum, Davide Mazzaccara, Dieu-Thu Le, Oliver Lemon, Alexander Koller
Venues:
LM Playschool | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
110–126
Language:
URL:
https://aclanthology.org/2026.lmplayschool-1.9/
DOI:
Bibkey:
Cite (ACL):
Sherzod Hakimov, Dimitrije Ristic, Philipp Sadler, Davide Mazzaccara, Sabrina McCallum, Filippo Momentè, Maria Manina, Karl Osswald, Dieu-Thu Le, Oliver Lemon, Alexander Koller, Raffaella Bernardi, Raquel Fernández, Mario Giulianelli, Alessandro Suglia, and David Schlangen. 2026. The LM Playschool Challenge 2026. In The Proceedings of Improving Language Models through Learning from Dialogue Game Interaction, pages 110–126, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
The LM Playschool Challenge 2026 (Hakimov et al., LM Playschool 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.lmplayschool-1.9.pdf
PDF Cite Search Fix data