Shogo Okada


pdf bib
Analyzing Differences in Subjective Annotations by Participants and Third-party Annotators in Multimodal Dialogue Corpus
Kazunori Komatani | Ryu Takeda | Shogo Okada
Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue

Estimating the subjective impressions of human users during a dialogue is necessary when constructing a dialogue system that can respond adaptively to their emotional states. However, such subjective impressions (e.g., how much the user enjoys the dialogue) are inherently ambiguous, and the annotation results provided by multiple annotators do not always agree because they depend on the subjectivity of the annotators. In this paper, we analyzed the annotation results using 13,226 exchanges from 155 participants in a multimodal dialogue corpus called Hazumi that we had constructed, where each exchange was annotated by five third-party annotators. We investigated the agreement between the subjective annotations given by the third-party annotators and the participants themselves, on both per-exchange annotations (i.e., participant’s sentiments) and per-dialogue (-participant) annotations (i.e., questionnaires on rapport and personality traits). We also investigated the conditions under which the annotation results are reliable. Our findings demonstrate that the dispersion of third-party sentiment annotations correlates with agreeableness of the participants, one of the Big Five personality traits.


pdf bib
Collection of Multimodal Dialog Data and Analysis of the Result of Annotation of Users’ Interest Level
Masahiro Araki | Sayaka Tomimasu | Mikio Nakano | Kazunori Komatani | Shogo Okada | Shinya Fujie | Hiroaki Sugiyama
Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)