NAISTeacher: A Prompt and Rerank Approach to Generating Teacher Utterances in Educational Dialogues

Justin Vasselli, Christopher Vasselli, Adam Nohejl, Taro Watanabe


Abstract
This paper presents our approach to the BEA 2023 shared task of generating teacher responses in educational dialogues, using the Teacher-Student Chatroom Corpus. Our system prompts GPT-3.5-turbo to generate initial suggestions, which are then subjected to reranking. We explore multiple strategies for candidate generation, including prompting for multiple candidates and employing iterative few-shot prompts with negative examples. We aggregate all candidate responses and rerank them based on DialogRPT scores. To handle consecutive turns in the dialogue data, we divide the task of generating teacher utterances into two components: teacher replies to the student and teacher continuations of previously sent messages. Through our proposed methodology, our system achieved the top score on both automated metrics and human evaluation, surpassing the reference human teachers on the latter.
Anthology ID:
2023.bea-1.63
Volume:
Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)
Month:
July
Year:
2023
Address:
Toronto, Canada
Editors:
Ekaterina Kochmar, Jill Burstein, Andrea Horbach, Ronja Laarmann-Quante, Nitin Madnani, Anaïs Tack, Victoria Yaneva, Zheng Yuan, Torsten Zesch
Venue:
BEA
SIG:
SIGEDU
Publisher:
Association for Computational Linguistics
Note:
Pages:
772–784
Language:
URL:
https://aclanthology.org/2023.bea-1.63
DOI:
10.18653/v1/2023.bea-1.63
Bibkey:
Cite (ACL):
Justin Vasselli, Christopher Vasselli, Adam Nohejl, and Taro Watanabe. 2023. NAISTeacher: A Prompt and Rerank Approach to Generating Teacher Utterances in Educational Dialogues. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 772–784, Toronto, Canada. Association for Computational Linguistics.
Cite (Informal):
NAISTeacher: A Prompt and Rerank Approach to Generating Teacher Utterances in Educational Dialogues (Vasselli et al., BEA 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.bea-1.63.pdf