Cleaning Dirty Books: Post-OCR Processing for Previously Scanned Texts

Allen Kim, Charuta Pethe, Naoya Inoue, Steve Skiena


Abstract
Substantial amounts of work are required to clean large collections of digitized books for NLP analysis, both because of the presence of errors in the scanned text and the presence of duplicate volumes in the corpora. In this paper, we consider the issue of deduplication in the presence of optical character recognition (OCR) errors. We present methods to handle these errors, evaluated on a collection of 19,347 texts from the Project Gutenberg dataset and 96,635 texts from the HathiTrust Library. We demonstrate that improvements in language models now enable the detection and correction of OCR errors without consideration of the scanning image itself. The inconsistencies found by aligning pairs of scans of the same underlying work provides training data to build models for detecting and correcting errors. We identify the canonical version for each of 17,136 repeatedly-scanned books from 58,808 scans. Finally, we investigate methods to detect and correct errors in single-copy texts. We show that on average, our method corrects over six times as many errors as it introduces. We also provide interesting analysis on the relation between scanning quality and other factors such as location and publication year.
Anthology ID:
2021.findings-emnlp.356
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2021
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Venues:
EMNLP | Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
4217–4226
Language:
URL:
https://aclanthology.org/2021.findings-emnlp.356
DOI:
10.18653/v1/2021.findings-emnlp.356
Bibkey:
Cite (ACL):
Allen Kim, Charuta Pethe, Naoya Inoue, and Steve Skiena. 2021. Cleaning Dirty Books: Post-OCR Processing for Previously Scanned Texts. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4217–4226, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Cleaning Dirty Books: Post-OCR Processing for Previously Scanned Texts (Kim et al., Findings 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.findings-emnlp.356.pdf