Annotating Agreement and Disagreement in Threaded Discussion

Jacob Andreas, Sara Rosenthal, Kathleen McKeown


Abstract
We introduce a new corpus of sentence-level agreement and disagreement annotations over LiveJournal and Wikipedia threads. This is the first agreement corpus to offer full-document annotations for threaded discussions. We provide a methodology for coding responses as well as an implemented tool with an interface that facilitates annotation of a specific response while viewing the full context of the thread. Both the results of an annotator questionnaire and high inter-annotator agreement statistics indicate that the annotations collected are of high quality.
Anthology ID:
L12-1650
Volume:
Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)
Month:
May
Year:
2012
Address:
Istanbul, Turkey
Editors:
Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
818–822
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2012/pdf/1095_Paper.pdf
DOI:
Bibkey:
Cite (ACL):
Jacob Andreas, Sara Rosenthal, and Kathleen McKeown. 2012. Annotating Agreement and Disagreement in Threaded Discussion. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 818–822, Istanbul, Turkey. European Language Resources Association (ELRA).
Cite (Informal):
Annotating Agreement and Disagreement in Threaded Discussion (Andreas et al., LREC 2012)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2012/pdf/1095_Paper.pdf