Dialogue-game benchmarks usually pair a model with itself, and the resulting self-play score is treated as a measure of that model’s ability. We test whether self-play scores measure partner-independent ability by comparing them with cross-play evaluations. Across nine Clembench games, we evaluate every possible seating of three popular open-weight models. The cross-play reading exposes additional model strengths and weaknesses that are not detectable in self-play. In four of nine games, a mixed pair of two different models beats the best self-play score; in another four cases, a model does better paired with itself than paired with the strongest available partner. Looking at the transcripts, two mechanisms explain most of this: some pairs succeed because one partner writes more detailed messages that keep the interaction on track, while some self-play scores are inflated because both copies of a model make the same mistake and agree by accident.

Existing machine translation (MT) metrics and discourse-focused evaluations primarily assess translation quality intrinsically, without measuring the downstream consequences of translation errors. In this work, we focus on extrinsic discourse evaluation of machine translation under two distinct regimes: static and interactive. Under the static regime, we propose an entity counting task as an illustrative probe of referential consistency in discourse. We show that high intrinsic MT quality does not reliably predict downstream discourse success and strong MT systems still produce referential inconsistencies. For the interactive regime, we study the goal-oriented multi-agent Welfare Diplomacy game as a probe of communication and coordination success over long horizons. We find that interaction-specific translation failures impact downstream coordination. Our results highlight goal-oriented translation-mediated environments as a viable framework for discourse-sensitive extrinsic MT evaluation.

This paper explores the impact of variable pragmatic competence on communicative success through simulating language learning and conversing between speakers and listeners with different levels of reasoning abilities. Through studying this interaction, we hypothesize that matching levels of reasoning between communication partners would create a more beneficial environment for communicative success and language learning. Our research findings indicate that learning from more explicit, literal language is advantageous, irrespective of the learner’s level of pragmatic competence. Furthermore, we find that integrating pragmatic reasoning during language learning, not just during evaluation, significantly enhances overall communication performance. This paper provides key insights into the importance of aligning reasoning levels and incorporating pragmatic reasoning in optimizing communicative interactions.

Asking for clarification is fundamental to effective collaboration. An interactive artificial agent must know when to ask a human instructor for more information in order to ascertain their goals. Previous work bases the timing of questions on supervised models learned from interactions between humans. Instead of a supervised classification task, we wish to ground the need for questions in the acting agent’s predictive uncertainty. In this work, we investigate if ambiguous linguistic instructions can be aligned with uncertainty in neural models. We train an agent using the T5 encoder-decoder architecture to solve the Minecraft Collaborative Building Task and identify uncertainty metrics that achieve better distributional separation between clear and ambiguous instructions. We further show that well-calibrated prediction probabilities benefit the detection of ambiguous instructions. Lastly, we provide a novel empirical analysis on the relationship between uncertainty and dialog history length and highlight an important property that poses a difficulty for detection.