Simulating Multimodal Multiparty Student Interactions with Large Language Models for Training Adaptive Pedagogical Agents

Tristan Maidment, Timothy Nokes-Malach, Diane Litman, Erin Walker, Adriana Kovashka


Abstract
Multiparty tutoring policies must regulate turn-taking and social attention, yet collecting the interaction data needed for reinforcement learning is expensive in educational settings. We introduce an LLM-based user simulator for triadic robot–student dialogue that represents gaze and gesture as inline stage directions. Combined with phase-tagged prompting (which delineates conversational structure) and per-turn persona reinforcement, this design allows the LLMs to generate interactions that preserve key discourse structure and controllable participation dynamics. We validate six simulator variants against a real student–robot corpus (12 student pairs) using metrics for semantic similarity, cue obedience, and persona consistency. While the simulator reaches strong semantic agreement with real transcripts, it also reveals a measurable sim-to-real style gap (in verbosity and grammatical complexity). The simulator enables the creation of an RL environment where the tutor selects factorized gaze and reference actions, learning to interact with simulated students. We compare online training (APPO) to offline learning (DQN). Off-policy evaluation on real logs shows that offline policies trained purely from synthetic data exceed online-trained policies under a doubly-robust estimator, and a verbal-only ablation yields unstable OPE due to heavy-tailed importance weights. These results support a scalable workflow for prototyping socially grounded multiparty tutoring policies while relaying where real-world coverage and OPE reliability remain bottlenecks.
Anthology ID:
2026.emnlp-main.129
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
2972–2986
Language:
URL:
https://aclanthology.org/2026.emnlp-main.129/
DOI:
Bibkey:
Cite (ACL):
Tristan Maidment, Timothy Nokes-Malach, Diane Litman, Erin Walker, and Adriana Kovashka. 2026. Simulating Multimodal Multiparty Student Interactions with Large Language Models for Training Adaptive Pedagogical Agents. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 2972–2986, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Simulating Multimodal Multiparty Student Interactions with Large Language Models for Training Adaptive Pedagogical Agents (Maidment et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.129.pdf