FeedFetcher: A Resilient Web Feed Downloader for Corpus Construction

Ondřej Herman, Jan Kraus, Vit Suchomel


Abstract
Building large-scale, timestamped monitor corpora requires robust and efficient tools for continuous web data acquisition. We present FeedFetcher, an open-source, lightweight yet resilient downloader designed to collect linguistic data from RSS/Atom web feeds. The tool enables continuous corpus updates by harvesting newly published web content with minimal downtime and high data integrity. Implemented in Rust for performance, memory safety, and scalable concurrency, FeedFetcher supports thousands of simultaneous connections while maintaining server politeness. The software is available under the GPL-3.0 license on https://github.com/ondra/feed_fetcher. In our setup, the entire workflow integrates FeedFetcher with downstream text-processing pipelines for tokenization, lemmatization, corpus compilation and deployment. The system is currently used to update monitor corpora in 64 languages, producing approximately two billion tokens per month. These corpora are available in Sketch Engine. We also describe methods for discovering new web feeds, combining manual exploration with automated extraction from large-scale web crawls to expand linguistic coverage. We demonstrate the system’s applicability through a time-based analysis of word-frequency change, showing how long-term accumulation of timestamped data supports the study of lexical dynamics and language evolution.
Anthology ID:
2026.lrec-1.558
Volume:
Proceedings of the Fifteenth Language Resources and Evaluation Conference
Month:
May
Year:
2026
Address:
Palma de Mallorca, Spain
Editors:
Stelios Piperidis, Núria Bel, Henk van den Heuvel, Nancy Ide, Simon Krek, Antonio Toral
Venue:
LREC
SIG:
Publisher:
ELRA Language Resource Association
Note:
Pages:
7014–7022
Language:
External URL:
https://lrec.elra.info/lrec2026-main-558
DOI:
10.63317/49txz3zreas2
Bibkey:
Cite (ACL):
Ondřej Herman, Jan Kraus, and Vit Suchomel. 2026. FeedFetcher: A Resilient Web Feed Downloader for Corpus Construction. In Proceedings of the Fifteenth Language Resources and Evaluation Conference, pages 7014–7022, Palma de Mallorca, Spain. ELRA Language Resource Association.
Cite (Informal):
FeedFetcher: A Resilient Web Feed Downloader for Corpus Construction (Herman et al., LREC 2026)
Copy Citation: