Evaluation of the KomParse Conversational Non-Player Characters in a Commercial Virtual World

Tina Kluewer, Feiyu Xu, Peter Adolphs, Hans Uszkoreit


Abstract
The paper describes the evaluation of the KomParse system. KomParse is a dialogue system embedded in a 3-D massive multiplayer online game, allowing conversations between non player characters (NPCs) and game users. In a field test with game users, the system was evaluated with respect to acceptability and usability of the overall system as well as task completion, dialogue control and efficiency of three conversational tasks. Furthermore, subjective feedback has been collected for evaluating the single communication components of the system such as natural language understanding. The results are very satisfying and promising. In general, both the usability and acceptability tests show that the tested NPC is useful and well-accepted by the users. Even if the NPC does not always understand the users well and expresses things unexpected, he could still provide appropriate responses to help users to solve their problems or entertain them.
Anthology ID:
L12-1419
Volume:
Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)
Month:
May
Year:
2012
Address:
Istanbul, Turkey
Editors:
Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
3535–3542
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2012/pdf/721_Paper.pdf
DOI:
Bibkey:
Cite (ACL):
Tina Kluewer, Feiyu Xu, Peter Adolphs, and Hans Uszkoreit. 2012. Evaluation of the KomParse Conversational Non-Player Characters in a Commercial Virtual World. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 3535–3542, Istanbul, Turkey. European Language Resources Association (ELRA).
Cite (Informal):
Evaluation of the KomParse Conversational Non-Player Characters in a Commercial Virtual World (Kluewer et al., LREC 2012)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2012/pdf/721_Paper.pdf