Quan Duong


pdf bib
An Unsupervised method for OCR Post-Correction and Spelling Normalisation for Finnish
Quan Duong | Mika Hämäläinen | Simon Hengchen
Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)

Historical corpora are known to contain errors introduced by OCR (optical character recognition) methods used in the digitization process, often said to be degrading the performance of NLP systems. Correcting these errors manually is a time-consuming process and a great part of the automatic approaches have been relying on rules or supervised machine learning. We build on previous work on fully automatic unsupervised extraction of parallel data to train a character-based sequence-to-sequence NMT (neural machine translation) model to conduct OCR error correction designed for English, and adapt it to Finnish by proposing solutions that take the rich morphology of the language into account. Our new method shows increased performance while remaining fully unsupervised, with the added benefit of spelling normalisation. The source code and models are available on GitHub and Zenodo.

pdf bib
TFW2V: An Enhanced Document Similarity Method for the Morphologically Rich Finnish Language
Quan Duong | Mika Hämäläinen | Khalid Alnajjar
Proceedings of the Workshop on Natural Language Processing for Digital Humanities

Measuring the semantic similarity of different texts has many important applications in Digital Humanities research such as information retrieval, document clustering and text summarization. The performance of different methods depends on the length of the text, the domain and the language. This study focuses on experimenting with some of the current approaches to Finnish, which is a morphologically rich language. At the same time, we propose a simple method, TFW2V, which shows high efficiency in handling both long text documents and limited amounts of data. Furthermore, we design an objective evaluation method which can be used as a framework for benchmarking text similarity approaches.