Children acquire language from far less input than language models, and social interaction is widely thought to be part of the reason. Dialogue games cannot fully replicate human interaction, but they offer a controllable environment for simulating situated language use and for studying how models learn from and through interaction. The LM Playschool Challenge builds on this idea, shifting language model training away from observational data towards multi-turn language use. Participants were free to post-train a language model of any size, using any data or method, with one objective in mind — to improve its performance on an unseen set of dialogue games. Our findings suggest that post-training on dialogue games effectively teaches game rules and structure and can support generalisation, especially for small models — the best system achieved gains of roughly seven and three times its baseline score on held-out instances of seen games and on unseen games, respectively. However, gains on unseen games were concentrated in a few games, and larger models improved mainly by achieving better completion rates. General ability was largely retained across submitted systems, owing to the fact that fine-tuning on dialogue games produces shorter responses — and in turn, trigger fewer formatting errors on static benchmarks. Nearly all submissions relied on supervised fine-tuning, often followed by preference optimisation, and none trained on the dense turn-level feedback offered by Playpen, leaving this an open direction for future iterations of the challenge.

Despite the ability of large language models (LLMs) to generate coherent comparative answers, automatic comparative question answering (CQA) remains challenging due to the absence of standardized evaluation criteria and the high resource demands of manual assessment. To address these problems, this paper proposes a comprehensive evaluation framework designed to assess the quality of CQA summaries using LLMs-as-a-Judge. We formulate 15 evaluation criteria for assessing comparative answers generated by various sources, including LLMs, human experts, and prior work. To capture a diverse range of comparative answers, LLM summaries were generated under various prompting scenarios. We evaluate the effectiveness of our framework using both human assessment and LLMs, demonstrating the consistency between automated and manual evaluations. Finally, we fine-tune Llama-3-8B-Instruct on a dataset generated from the best-performing CQA models in our evaluation benchmark to assess whether open-source models can replace closed-source LLMs. All used data, code, and evaluation results are publicly available.

Comparative Question Answering (CQA) lies at the intersection of Question Answering, Argument Mining, and Summarization. It poses unique challenges due to the inherently subjective nature of many questions and the need to integrate diverse perspectives. Although the CQA task can be addressed using recently emerged instruction-following Large Language Models (LLMs), challenges such as hallucinations in their outputs and the lack of transparent argument provenance remain significant limitations.To address these challenges, we construct a manually curated dataset comprising arguments annotated with their relevance. These arguments are further used to answer comparative questions, enabling precise traceability and faithfulness. Furthermore, we define explicit criteria for an “ideal” comparison and introduce a benchmark for evaluating the outputs of various Retrieval-Augmented Generation (RAG) models with respect to argument relevance. All code and data are publicly released to support further research.