Large-scale web corpora are valuable resources for natural language processing (NLP), linguistic research, and language model training, particularly for digitally low-resource languages such as Persian. However, web-derived text often contains document-level noise, even after quality-filtering pipelines. We present HPLT3 Persian Refined (HPLT3-PerRef), a refined Persian web corpus containing 105.9 billion tokens and 102.37 million documents, obtained by applying a supervised quality-filtering pipeline to the Persian subset of HPLT3. We manually annotated 6,144 Persian web documents and trained a multilingual transformer-based classifier to distinguish high- and low-quality documents. The classifier removed approximately 20% of the original corpus while preserving broad domain coverage and complements HPLT3’s heuristic-based Web Document Score (WDS) by identifying additional low-quality documents. We publicly release the corpus, the trained classifier, and the complete source code.

Web-scale text corpora provide NLP researchers and corpus linguists with access to vast quantities of language data, but efficiently identifying the subsets of a large corpus that are relevant to a particular research aim remains a significant challenge. Existing approaches, including keyword search and task-specific classifiers, are often either too restrictive or depend on costly retraining for each new research question. We explore and evaluate the feasibility of a task-agnostic, LLM-driven descriptor schema for annotating web documents, using LLMs to generate natural-language labels that capture stylistic, tonal, topical, and structural properties of texts. Unlike prior work that emphasizes broad categorization or quality filtering, our approach yields fine-grained and interpretable descriptors designed to support flexible downstream applications. We evaluate the approach from multiple angles and show that descriptor-based annotation can enable flexible document retrieval while also exposing challenges related to schema harmonization and descriptor specificity.

Web registers, the situationally defined varieties of online text, are almost always annotated at document level. A document, however, may contain multiple such varieties. We label 80,000 documents from the HPLT web corpus at segment level in seven languages using the open-weight Gemma 4, reaching Cohen’s κ=0.73 against 1,878 human-annotated segments, comparable to previously reported document-level agreement. We find that most documents have two or more registers, and that they recur in similar positions and combinations across the studied languages. Yet on average, a single register accounts for 73% of the labeled text, and a quarter to a third of segments receive no label. Document-level labels are therefore mostly correct but incomplete. Multi-dimensional analysis shows that the segment-level labels correspond to linguistic differences. In all seven languages, registers are ordered the same way, from informational to involved text.

Transformer-based web register classifiers are widely used to provide metadata for web crawls and to focus web-data sampling on specific register classes, such as news articles or advertisements. However, it remains unclear what these classifiers actually learn: do they detect pervasive linguistic features characteristic of registers, or do they rely on structural cues more indicative of genre? We examine this question through two hypotheses: the text-macrostructure hypothesis proposes that classifiers rely on structural features more typical of genres, whereas the feature-density hypothesis predicts that they identify typically register-associated features. Using English, Finnish, and French data from HPLT 3.0, we segment documents into thirds and compare classifier predictions for full documents and segments through five complementary analyses: main-label agreement, cosine similarity, confusion profiles, SACX keyword-weight correlations, and SACX keyword stability. As a more direct control, we compare predictions for a sample of untruncated documents with predictions for the same documents after their sentence order has been randomised. Top-label assignments and prediction vectors remain highly similar after shuffling, although the positional gradients are attenuated, indicating that sentence order contributes to classifier behaviour without determining its broader prediction profile. Overall, the results are more consistent with the feature-density hypothesis, although both hypotheses retain explanatory value.

In this paper, we use LLM perplexity as a measure to assess Greek dialectal distance. We test seven models on Standard Modern Greek (SMG) and eight dialects, namely Heptanesian, Cypriot, Maniot, Pontic, Northern, Cretan, Tsakonian, and Griko. Using samples of 5k, 15k, and 25k tokens from the GRDD+ corpus for each variety, we find a consistent dialect ranking across models, with Heptanesian closest to SMG, and Griko most distant (perplexity ratio 3.6–14.5× depending on model). These results are largely in agreement with theoretical dialectological knowledge. For example, Tsakonian consistently appears distant in all measures, reflecting its status as the sole Doric descendant, while Heptanesian appears closer by all metrics, pointing to its status as one of the dialects used to shape the official variety. Perplexity correlates strongly with Bits Per-Character (mean r = 0.94) and Normalized Compression Distance (mean r = 0.87, range 0.76–0.93), providing support for its use as a dialectometric tool. However, a number of important confounds are also found. First, tokenization effects compress Llama 2’s perplexity range. Second, genre artifacts seem to inflate the results for Cretan. Third, potential training data contamination likely reduces perplexity for Cypriot and Pontic. Lastly, we find that Greek-specific models like Meltemi and Krikri do not consistently outperform general models.

Despite recent advances in automatic web register (genre) labeling and its applications to web-scale datasets and LLM development, the effectiveness of these tools for digitally lowresource languages remains unclear. This study introduces ParsCORE, the first largescale collection of Persian web registers (genres), and evaluates deep learning models for register classification and keyword analysis across major registers. Using 2,000 humanannotated documents, the models achieved a micro F1-score of 0.76. The findings provide a foundation for future research on the linguistic and cultural specificities of Persian registers.

Nearly all studies on web registers—online text varieties associated with characteristic social contexts and linguistic features—use full documents as the unit of analysis. However, web documents often contain sections in different registers. A cooking blog, for instance, may combine personal storytelling, recipe instructions, user comments, and promotional text within a single URL. This internal variation raises doubts about the validity of document level register labeling. In this paper, we propose an LLM-based approach that identifies register homogeneous segments within documents and apply it to a 10,000-document English sample from HPLT 3.0. We show that segmentation addresses persistent problems in register analysis, including low inter-annotator agreement and category fuzziness. Strikingly, it also reveals that most web documents contain more than one register, making register mixing the norm rather than the exception on the web.

The development of high-quality multilingual models depends on well-curated multilingual datasets, which are typically obtained through filtering large web-crawled corpora. Such filtering approaches must be effective across a wide range of languages in order to improve model quality proportionally across the entire linguistic space. In recent years multiple Natural Language Processing (NLP) approaches for data processing and Large Language Model (LLM) training were developed. However, the most significant progress has been made for rich-resource languages such as English and Chinese, leaving multilingual methods largely underexplored. In this work, we present data selection strategies based on Internet registers (genres) and Web Document Score (WDS). We study 15 European languages utilizing HPLT 3.0 datasets. We train and evaluate four small generative language models per language, and compare how specific registers and WDS affect model training in separate and combined approaches. We use five common evaluation tasks covering models’ linguistic, knowledge-based, and reasoning skills. We analyze the impact of the studied strategies on each language, their average performance, and show how combining register-based selection with WDS improves evaluation performance. These results indicate that both registers and WDS can generalize across multiple languages, offering a new path towards multilingual data filtering.

Data quality is crucial for training Large Language Models (LLMs). Traditional heuristic filters often miss low-quality text or mistakenly remove valuable content. In this paper, we introduce an LLM-based line-level filtering method to enhance training data quality. We use GPT-4o mini to label a 20,000-document sample from FineWeb at the line level, allowing the model to create descriptive labels for low-quality lines. These labels are grouped into nine main categories, and we train a DeBERTa-v3 classifier to scale the filtering to a 10B-token subset of FineWeb. To test the impact of our filtering, we train GPT-2 models on both the original and the filtered datasets. The results show that models trained on the filtered data achieve higher accuracy on the HellaSwag benchmark and reach their performance targets faster, even with up to 25% less data. This demonstrates that LLM-based line-level filtering can significantly improve data quality and training efficiency for LLMs. We release our quality-annotated dataset, FinerWeb-10BT, and the codebase to support further work in this area.

This study introduces a novel method for analyzing register variation in web texts through classification-based register segmentation. While traditional text-linguistic register analysis treats web documents as single units, we present a recursive binary segmentation approach that automatically identifies register shifts within web documents without labeled segment data, using a ModernBERT classifier fine-tuned on full web documents. Manual evaluation shows our approach to be reliable, and our experimental results reveal that register segmentation leads to more accurate register classification, helps models learn more distinct register categories, and produces text units with more consistent linguistic characteristics. The approach offers new insights into documentinternal register variation in online discourse.

We describe the progress of the High Performance Language Technologies (HPLT) project, a 3-year EU-funded project that started in September 2022. We focus on the up-to-date results on the release of free text datasets derived from web crawls, one of the central objectives of the project. The second release used a revised processing pipeline, and an enlarged set of input crawls. From 4.5 petabytes of web crawls we extracted 7.6T tokens of monolingual text in 193 languages, plus 380 million parallel sentences in 51 language pairs. We also release MultiHPLT, a cross-combination of the parallel data, which produces 1,275 pairs, as well as releasing the containing documents for all parallel sentences in order to enable research in document-level MT. We report changes in the pipeline, analysis and evaluation results for the second parallel data release based on machine translation systems. All datasets are released under a permissive CC0 licence.

Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual monolingual and parallel corpora, extending prior work of the HPLT project. The monolingual portion of the data contains 8T tokens covering 193 languages, while the parallel data contains 380M sentence pairs covering 51 languages. We document the entire data pipeline and release the code to reproduce it. We provide extensive analysis of the quality and characteristics of our data. Finally, we evaluate the performance of language models and machine translation systems trained on HPLT v2, demonstrating its value.

Web-scale corpora present valuable research opportunities but often lack detailed metadata, making them challenging to use in linguistics and social sciences. This study tackles this problem by exploring automatic methods to classify web corpora into specific categories, focusing on text registers such as Interactive Discussion and literary genres such as Politics and Social Sciences. We train two machine learning models to classify documents from the large web-crawled OSCAR dataset: a register classifier using the multilingual, manually annotated CORE corpus, and a genre classifier using a dataset based on Kindle US&UK. Fine-tuned from XLM-R Large, the register and genre classifiers achieved F1-scores of 0.74 and 0.70, respectively. Our analysis includes evaluating the distribution of the predicted text classes and examining the intersection of genre-register pairs using topic modelling. The results show expected combinations between certain registers and genres, such as the Lyrical register often aligning with the Literature & Fiction genre. However, most registers, such as Interactive Discussion, are divided across multiple genres, like Engineering & Transportation and Politics & Social Sciences, depending on the discussion topic. This enriched metadata provides valuable insights and supports new ways of studying digital cultural heritage.

In corpus linguistics, registers–language varieties suited to different contexts–have traditionally been defined by their situations of use, yet recent studies reveal significant situational variation within registers. Previous quantitative studies, however, have been limited to English, leaving this variation in other languages largely unexplored. To address this gap, we apply a quantitative situational analysis to a large multilingual web register corpus, using large language models (LLMs) to annotate texts in English, Finnish, French, Swedish, and Turkish for 23 situational parameters. Using clustering techniques, we identify six situational text types, such as “Advice”, “Opinion” and “Marketing”, each characterized by distinct situational features. We explore the relationship between these text types and traditional register categories, finding partial alignment, though no register maps perfectly onto a single cluster. These results support the quantitative approach to situational analysis and are consistent with earlier findings for English. Cross-linguistic comparisons show that language accounts for only a small part of situational variation within registers, suggesting registers are situationally similar across languages. This study demonstrates the utility of LLMs in multilingual register analysis and deepens our understanding of situational variation within registers.

This paper introduces new models designed to improve the morpho-syntactic parsing of the five largest Latin treebanks in the Universal Dependencies (UD) framework. First, using two state-of-the-art parsers, Trankit and Stanza, along with our custom UD tagger, we train new models on the five treebanks both individually and by combining them into novel merged datasets. We also test the models on the CIRCSE test set. In an additional experiment, we evaluate whether this set can be accurately tagged using the novel LASLA corpus (https://github.com/CIRCSE/LASLA). Second, we aim to improve the results by combining the predictions of different models through an atomic morphological feature voting system. The results of our two main experiments demonstrate significant improvements, particularly for the smaller treebanks, with LAS scores increasing by 16.10 and 11.85%-points for UDante and Perseus, respectively (Gamba and Zeman, 2023a). Additionally, the voting system for morphological features (FEATS) brings improvements, especially for the smaller Latin treebanks: Perseus 3.15% and CIRCSE 2.47%-points. Tagging the CIRCSE set with our custom model using the LASLA model improves POS 6.71 and FEATS 11.04%-points respectively, compared to our best-performing UD PROIEL model. Our results show that larger datasets and ensemble predictions can significantly improve performance.

This article introduces a resource-efficient method for developing question-answer (QA) datasets by extracting QA pairs from web-scale data using machine learning (ML). Our method benefits from recent advances in web register (genre) identification and consists of two ML steps with an additional post-processing step. First, using XLM-R and the multilingual CORE web register corpus series with categories such as QA Forum, we train a multilingual classifier to retrieve documents that are likely to contain QA pairs from web-scale data. Second, we develop a NER-style token classifier to identify the QA text spans within these documents. To this end, we experiment with training on a semi-synthetic dataset built on top of the English LFQA, a small set of manually cleaned web QA pairs in English and Finnish, and a Finnish web QA pair dataset cleaned using ChatGPT. The evaluation of our pipeline demonstrates its capability to efficiently retrieve a substantial volume of QA pairs. While the approach is adaptable to any language given the availability of language models and extensive web data, we showcase its efficiency in English and Finnish, developing the first open, non-synthetic and non-machine translated QA dataset for Finnish – Turku WebQA – comprising over 200,000 QA pairs.