A Large-Scale Comparison of Historical Text Normalization Systems

Marcel Bollmann


Abstract
There is no consensus on the state-of-the-art approach to historical text normalization. Many techniques have been proposed, including rule-based methods, distance metrics, character-based statistical machine translation, and neural encoder–decoder models, but studies have used different datasets, different evaluation methods, and have come to different conclusions. This paper presents the largest study of historical text normalization done so far. We critically survey the existing literature and report experiments on eight languages, comparing systems spanning all categories of proposed normalization techniques, analysing the effect of training data quantity, and using different evaluation methods. The datasets and scripts are made publicly available.
Anthology ID:
N19-1389
Volume:
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)
Month:
June
Year:
2019
Address:
Minneapolis, Minnesota
Editors:
Jill Burstein, Christy Doran, Thamar Solorio
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3885–3898
Language:
URL:
https://aclanthology.org/N19-1389
DOI:
10.18653/v1/N19-1389
Bibkey:
Cite (ACL):
Marcel Bollmann. 2019. A Large-Scale Comparison of Historical Text Normalization Systems. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3885–3898, Minneapolis, Minnesota. Association for Computational Linguistics.
Cite (Informal):
A Large-Scale Comparison of Historical Text Normalization Systems (Bollmann, NAACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/N19-1389.pdf
Software:
 N19-1389.Software.zip
Poster:
 N19-1389.Poster.pdf
Code
 coastalcph/histnorm +  additional community code