Automatic parsing as an efficient pre-annotation tool for historical texts

Hanne Martine Eckhoff, Aleksandrs Berdičevskis


Abstract
Historical treebanks tend to be manually annotated, which is not surprising, since state-of-the-art parsers are not accurate enough to ensure high-quality annotation for historical texts. We test whether automatic parsing can be an efficient pre-annotation tool for Old East Slavic texts. We use the TOROT treebank from the PROIEL treebank family. We convert the PROIEL format to the CONLL format and use MaltParser to create syntactic pre-annotation. Using the most conservative evaluation method, which takes into account PROIEL-specific features, MaltParser by itself yields 0.845 unlabelled attachment score, 0.779 labelled attachment score and 0.741 secondary dependency accuracy (note, though, that the test set comes from a relatively simple genre and contains rather short sentences). Experiments with human annotators show that preparsing, if limited to sentences where no changes to word or sentence boundaries are required, increases their annotation rate. For experienced annotators, the speed gain varies from 5.80% to 16.57%, for inexperienced annotators from 14.61% to 32.17% (using conservative estimates). There are no strong reliable differences in the annotation accuracy, which means that there is no reason to suspect that using preparsing might lower the final annotation quality.
Anthology ID:
W16-4009
Volume:
Proceedings of the Workshop on Language Technology Resources and Tools for Digital Humanities (LT4DH)
Month:
December
Year:
2016
Address:
Osaka, Japan
Venues:
LT4DH | WS
SIG:
Publisher:
The COLING 2016 Organizing Committee
Note:
Pages:
62–70
Language:
URL:
https://aclanthology.org/W16-4009
DOI:
Bibkey:
Cite (ACL):
Hanne Martine Eckhoff and Aleksandrs Berdičevskis. 2016. Automatic parsing as an efficient pre-annotation tool for historical texts. In Proceedings of the Workshop on Language Technology Resources and Tools for Digital Humanities (LT4DH), pages 62–70, Osaka, Japan. The COLING 2016 Organizing Committee.
Cite (Informal):
Automatic parsing as an efficient pre-annotation tool for historical texts (Eckhoff & Berdičevskis, 2016)
Copy Citation:
PDF:
https://aclanthology.org/W16-4009.pdf