Document conversion and import are a large but under-studied source of real-world noisy text, distinct from the social-media noise usually studied in normalization. We study missing-space artifacts in a low-resource, Arabic-script setting: an Urdu Qur’anic tafseer exported from InPage .inp to .txt and then to JSON/markdown, a pipeline that fuses adjacent words into single tokens (e.g., اورقرآن for اور قرآن, "and the Qur’an"). Restoring spaces is hard: Urdu’s cursive Arabic script does not require spaces at word boundaries, and the text interleaves Urdu prose with Qur’anic Arabic, proper names, and markdown, so a careless split can corrupt preservation-sensitive content. We cast this as precision-first normalization and present a confidence-gated pipeline that proposes only space-insertion edits, verifies a character-preservation invariant, and holds protected regions out of automatic correction. On a 1,000-row silver-labeled sample, the high-confidence auto-accept operating point reaches 96.9% strict precision on decisive cases, against 30.5% for an ungated function-word splitter and 39.7% for a frequency-only baseline, with 0 character-altering edits. This suggests that confidence gating is necessary before applying whitespace restoration to preservation-sensitive Urdu–Arabic text. As our labels are model-assisted silver rather than expert-gold, we treat the precision figures as provisional and release derived artifacts for reproducibility.

Emotion analysis systems are expected by users to provide reliable results. Optimizing their robustness, in addition to accuracy, is particularly challenging in noisy environments, such as social media data or other forms of naturally expressed language. Our hypothesis is that robustness can be improved by guiding a language model to pay attention to multiple emotion signals in the input text. Therefore, requesting labels according to multiple emotion theories may improve robustness. To study this hypothesis, we run permutation experiments on input instances to test the robustness of LLM configurations based on basic emotions only, and jointly with valence–arousal, and cognitive appraisals. We find that adding valence–arousal and cognitive appraisal along with basic emotion instructions to the emotion classification improves robustness, with the effect being most pronounced for adding appraisals. In a further analysis, we find that this improvement is less susceptible to word order permutations and more to random word substitutions and deletions. These differences hold across multiple LLMs studied in our work.

Event detection (ED) systems are typically evaluated on clean, curated text, leaving their robustness to real-world noise largely unexplored, particularly for low-resource languages such as Bangla. We introduce a generalized Bangla news event ontology and a benchmark comprising 9,979 annotated sentences across 40 event subtypes, spanning clean news text, real-world Automatic Speech Recognition (ASR) transcripts, and orthographically corrupted text. We systematically evaluate fine-tuned encoder-only models (BanglaBERT and XLM-R) alongside instruction-tuned decoder-only large language models (Llama 3 and Gemma 3). Our results reveal a clear architectural trade-off: encoder models achieve higher performance on clean text but degrade substantially under noise, whereas decoder-only LLMs are markedly more robust, particularly when event triggers are corrupted. We further show that embedding annotation guidelines during instruction tuning establishes a higher performance baseline on noisy text but yields inconsistent reductions in performance degradation across noisy conditions. Finally, model scaling consistently improves the robustness of decoder-only LLMs, while combined training on clean and noisy data serves as an effective regularization strategy that disproportionately benefits encoder architectures, significantly narrowing the robustness gap.

Identifying the targets of emotional words and phrases in crisis situations, particularly health-related crises, is essential for understanding public concerns across cultural and linguistic contexts. We propose CrisisKD, a five-stage teacher–student knowledge distillation framework for aspect-level sentiment and emotion analysis on unannotated social media data. A teacher large language model generates aspect-level labels and reasoning traces to supervise a smaller student model across five tasks: aspect extraction, syntactic parsing, opinion extraction, sentiment classification, and emotion classification. Using this framework, we construct and release a dataset containing 50,615 aspect-level annotations, along with the annotation pipeline and fine-tuned student models as open-source resources. The resulting student model supports end-to-end aspect-based sentiment analysis and emotion detection at substantially lower inference cost than the teacher, making the approach suitable for practitioners with limited access to annotated data and large-scale computational resources. Because CrisisKD requires only raw text and a dependency parser, the framework is architecturally portable to other crisis domains and to languages supported by suitable multilingual parsers.

Large language models (LLMs) are increasingly utilized to generate explanations of complex phenomena, yet existing evaluation frameworks assess quality extensively through semantic similarity to human references, a measure blind to discourse organization. We conduct a computational study of 300 ELI5 question pairs, comparing Llama-3.1 8B, Gemma-2 9B, and Mistral-7B against human-written responses using PDTB-grounded connective analysis across five relation types. All three models exhibit elaborative dominance, i.e. Elaborative connectives account for 56-66% of discourse markers versus 38% in human writing, with Causal, Contrastive, and Conditional relations systematically suppressed. This 18-28 percentage point gap persists after connective sense disambiguation (11-20 pp), ruling out measurement artifact. Across three semantic similarity metrics, PDTB connective noise explains less than 8% of shared variance (r²<0.08), confirming it captures evaluation signal invisible to existing approaches. We establish discourse-structural noise as a distinct, measurable evaluation dimension for LLM-generated explanations.

Survey methodology requires that equivalent question formulations elicit stable responses from the same respondent, regardless of how the Likert-type answer scale is presented. As LLMs are increasingly deployed as synthetic survey respondents, a critical but underexplored question arises: do they satisfy this basic parallel-forms reliability standard when answer scales vary in wording, cardinality, polarity, or option order? We extend the OpinionQA into two stress-test benchmarks, 1,235 subjective questions each paired with five reworded scale variants and six positional permutations, evaluated across four prompt-label formats. Using a four-metric consistency framework, we benchmark eight open-weight models from four families in both base and instruction-tuned configurations. Three findings emerge. First, base models fail basic reliability requirements: they exhibit severe mechanical instability driven by primacy bias, label sensitivity, and formatting failures, including full polarity inversions for the same question. Second, instruction tuning acts as a reliability filter, reducing response variance by approximately 50% and enabling models to treat semantically equivalent scales as equivalent. Third, for instruction-tuned models under randomized option ordering, removing explicit labels paradoxically increases consistency by eliminating symbol-binding conflicts. These findings establish that response reliability is an engineered property of LLMs requiring both alignment training and deliberate prompt design, with concrete implications for survey-like MCQA evaluations.

As large language models (LLMs) are increasingly used to simulate human behavior, their operational validity is typically assessed through accuracy alone, implicitly treating human behavior as deterministic. We argue that on noisy digital trace data, a model’s performance depends not only on predicting the correct outcome but on its calibration. We present a case study testing whether the token probabilities of open-weight LLMs can serve as confidence estimates in a Next Like Prediction task: predicting how a user of the Luxembourgish news platform RTL.lu rates a comment, conditioned on their prior interactions. Evaluating Ministral 3 (3B, 8B), we find that, with in-context learning only, the tested models are overconfident and are outperformed on accuracy and calibration by a simple statistical baseline. Accuracy and calibration do not move together: scaling from 3B to 8B improves accuracy while degrading calibration, and longer behavioral histories raise accuracy while confidence declines. Task-specific supervised fine-tuning acts as an effective calibration intervention, reducing Expected Calibration Error from 0.2 to 0.006 while improving accuracy from 0.536 to 0.758. We conclude that confidence estimation should be evaluated alongside accuracy to improve validity when using LLM-based agents.

Large language models (LLMs) offer disaster risk management (DRM) practitioners a promising “plug-and-play” solution for information extraction on online media data. However, they can struggle with the strict precision required for identifying semantically and structurally complex disaster causes. We evaluated how off-the shelf LLMs extract causal information from flood-related texts and found a strong recall bias. To diagnose these errors, our primary contribution is a comprehensive, human-annotated qualitative error taxonomy using inductive and deductive coding. We identified specific linguistic traps, i.e., impact descriptions, background context, and subjective opinions, trigger over-extraction, indicating models have a patterned failure. As an exploratory second step, we apply this taxonomy to design an ablation study of prompt-based interventions using definitions and exclusions. Our results show that our approach successfully shifts models along the precision–recall trade-off, i.e., curbing over-extraction, improving precision and domain-specific F0.5-score for recall-biased models. We demonstrate that by leveraging qualitative error analysis, domain experts can not only characterize and refine off-the-shelf LLMs, but also navigate the tradeoffs of prompting.

Traditional Named Entity Recognition (NER) metrics rely on syntactic span and type matching. While this may be appropriate if NER is framed as a sequence labeling task, the advent of generative models allows for alternative approaches, where model outputs are not extracted, but generated. In this perspective, generative NER calls for more semantically-aware scoring mechanisms. Generative models may produce valid aliases, paraphrases, or type variants that differ from the annotated gold, while also producing unsupported predictions. We propose a taxonomy of such cases and introduce an embedding-based parametric method to evaluate generative NER based on three metrics: soft precision, soft recall and unsupported rate. We validate this framework using standard and noisy datasets, across state-of-the-art systems such as GPT and GLiNER alongside older encoder-based models. We find that our metrics mostly agree with traditional ones on better-established, less noisy benchmarks, but differ in more recent, noisier ones, and can capture edge cases that arise from free-form generation.

Although human variability has traditionally been the primary source of annotation noise, the increasing use of Large Language Models (LLMs) for automated labeling is shifting this landscape toward systematic, machine-generated label noise. This challenge is particularly severe in Subjectivity Detection (SD), where the behavior of automated errors remains largely unexamined. This study characterizes the extent and linguistic nature of automated annotation noise in Spanish SD. Errors are evaluated as omissions (under-generation) or additions (over-generation) across six subjectivity dimensions: Doubt, Certainty, Specificity, Non-specificity, Distancing, and Participation. Using an expert-annotated human gold standard as a reference, a traditional machine learning control is benchmarked against three state-of-the-art LLM families (GPT, Gemini, and Llama). The results reveal a clear architectural divide: while the traditional ML framework predominantly generates omission errors (63.99%), only detecting markers it was explicitly programmed to recognize, LLMs like Gemini and Llama lean heavily toward addition errors (exceeding 51%), guessing and hallucinating markers instead of leaving uncertain cases unlabeled. Although Gemini minimizes overall noise, achieving F1-scores up to 0.96 in specific dimensions, Certainty remains highly susceptible to errors, showing a 53.96% error rate in the traditional ML control. Ultimately, this diagnostic evaluation of LLMs as annotators in this specific subjective task reveals that automated systems introduce systematic label noise, demonstrating that human-in-the-loop participation is indispensable to resolve these errors and ensure reliable dataset curation.

Since the proliferation of LLMs, there have been concerns about their misuse for harmful content creation and dissemination. Recent studies justify such fears, providing evidence of LLM vulnerabilities and high misuse potential. It has been shown that humans struggle to reliably distinguish high-quality machine-generated text from authentic human-written text. Therefore, it is crucial to develop automated means to accurately detect machine-generated content. While multiple detectors of machine-generated texts have been researched so far, a known issue is their low generalizability to unseen generators, domains or languages. This work addresses the problem by proposing a robustness-oriented LLM training methodology for the detection task, making the detectors more robust against obfuscation and more generalizable to out-of-distribution data. The fixed-size ablation study confirms that the gains are not simply a function of more data.

Language models are dependent on the character sets of the languages seen during training. Some languages use large character sets, like Chinese and Korean, while others are very small, like English. When the input during inference diverges from the trained character set, it can lead to unexpected behavior. While many researchers have looked at using this weakness for adversarial attacks, these studies typically focus on extreme cases that lead to model failure. But we don’t have a strong understanding of how brittle models are to more casual and routine character substitutions. In fact, many ‘known’ characters have Unicode lookalikes that users enjoy using without malicious purposes. This paper studies the fragility of models when faced with such character noise. When input contains innocuous character substitutions, do models robustly handle the distortion? How does the tokenizer and overall model size affect performance? This paper presents analyses of tokenizers and a series of fragility tests to answer these questions. We find that tokenization is not the only stumbling block, and neither is a larger model size the assured solution. We present a formal benchmark, publicly release a Fragility Scorer, and describe our experimental results on a variety of open-source models. Implications and suggestions for improving fragility are then presented.

We investigate the use of Large Language Models (LLMs) in multi-author style change detection. Using off-the-shelf LLMs or supervised fine-tuning solely within the target domain fails to yield effective features; for this reason, we propose cross-domain transfer learning with LLM-based embeddings fine-tuned for authorship verification. We fine-tune Mistral-7B embedding model and compare it with DeBERTa, used in the the state-of-the-art (SOTA) systems in PAN-25 multi-author writing style analysis challenge. Results show that DeBERTa and Mistral excel in different user-generated text scenarios. We exploit this diversity by combining both embeddings, leading to more effective style change detection.

Transformer-based models for Named Entity Recognition (NER) achieve strong performance on clean benchmarks, yet their reliability in real-world settings remains insufficiently understood. In practical applications, NER systems are frequently exposed to degraded input originating from optical character recognition, automatic speech recognition, and user-generated text. We address this gap by deriving a taxonomy of data quality perturbations from a systematic literature review, consolidating fragmented prior work into a unified framework. Guided by this taxonomy, we conduct controlled experiments evaluating six widely used transformer architectures under varying perturbation types, intensities, and training configurations. Our results show that syntactic perturbations cause the most severe performance degradation across models, while architectural features such as character-level processing and disentangled attention confer specific robustness advantages. Furthermore, we demonstrate that targeted training with perturbed data can recover more than half of the lost performance with minimal impact on clean-data accuracy.

Duplicate content is widely used to study coordinated behavior in social media information operations (IOs), but not all repetition provides equally meaningful evidence of coordination. Generic, reusable, or low-information posts may create noisy account-account links when projected into coordination graphs. We study this problem using 187,000 English-language tweets from six Russian Twitter Information Operations datasets. We introduce a generic/non-generic distinction for duplicate campaigns, label tweets using an LLM-assisted protocol with independent human validation, and train supervised classifiers over sentence embeddings to scale the labels. We construct duplicate campaigns using lexical similarity and two embedding-based methods. Generic campaigns are rare under lexical matching but account for nearly 39% of campaigns detected by embedding-based methods. Restricting graphs to non-generic campaigns reduces graph size and the largest connected component while increasing density, suggesting a smaller but more focused coordination structure. These findings show that duplicate-based coordination analysis should consider both textual similarity and semantic specificity.

Personalized emotion analysis aims to account for individual differences in emotional expression by adapting predictions to each writer.Existing personalization methods typically require dedicated feature extraction modules or modifications to model architectures, increasing model complexity. In this paper, we propose a prompt-based personalization framework for emotion analysis that incorporates writer information into the input prompt of a large language model (LLM), without requiring fine-tuning or architectural changes.Specifically, we investigate two complementary types of writer information: personality traits representing stable writer characteristics and post history representing writer-specific patterns of emotional expression. Experimental results on the WRIME corpus in Japanese show that incorporating temporally close post history consistently improves emotion intensity estimation across multiple LLMs. Personality information alone provides limited improvements, but yields additional gains for several models when combined with post history. These findings suggest that prompt-based personalization is a practical alternative to architecture-level personalization for LLM-based emotion analysis.

Adapting large language models to an individual author’s style from a few examples is challenging, and scientific writing sharpens the difficulty: formal conventions leave little surface variation, and authors write about their own topics, so extracted “style” easily entangles with content. We study style-conditioned abstract generation from a few example abstracts per author and propose three methods: (1) contrastive activation steering, (2) a network that predicts steering vectors, and (3) a hypernetwork that predicts LoRA adapters. We find a consistent trade-off between style imitation and output quality: fine-tuning buys most of the available style signal but forfeits fluency, while the hypernetwork achieves the best trade-off on both seen and unseen authors. Our steering operates at author level, contrasting an author’s abstracts against style-neutral generations for the same content. This holds topic fixed, removes the need for a predefined style inventory, and outperforms inventory-based steering. Moreover, our analyses demonstrate that manually extracted and predicted steering vectors are near-orthogonal yet score comparably, indicating that style conditioning here can admit at least two unrelated directions rather than requiring one particular axis.

Chat assistants increasingly ask large language models to turn free-form user prose into structured, executable diagrams. These requests are noisy user-generated text: informal, hedged, jargon-mixed, incomplete, and often describing the target only indirectly—in workflow language rather than diagram vocabulary. Evaluation is hard because many distinct diagrams satisfy the same noisy request, making reference-based metrics too rigid where no ground truth exists, while reference-free LLM judges give weak, hard-to-reproduce diagnostics. We introduce QueryDiagramEval (QDE), a reference-free protocol scoring diagram generation directly against raw, noisy queries via two levels: deterministic compilation gates (syntax, rendering, diagram-type correctness), then a semantic-fidelity layer where a multimodal auditor is grounded in type-specific structural facts parsed from the code. On 255 annotated predictions, QDE aligns strongly with holistic human ratings (𝜌=0.715), outperforming an ungrounded LLM judge. We also release MermaidBench, a source-grounded benchmark of 1,150 query–gold pairs over 11 diagram types, four query-difficulty tiers, and a multi-turn track, with the gold held fixed to isolate robustness to input-side noise. Furthermore, we conduct experiments on eight frontier-model and harnesses to show that the our agentic framework reaches near-gold quality while failures concentrate in topology preservation, label grounding, and type-specific syntax.

Small-scale Large Language Models (LLMs) natively default to literal semantic interpretations, making few-shot irony detection a persistent challenge in noisy, user-generated text. In this study, we show that, despite this limitation, inference from compact LLMs can still be effectively harnessed for this task. We introduce the Robust Dual-Signal (RDS) Fusion framework, a hybrid neuro-symbolic architecture that utilizes a compressed Chain-of-Thought (CoT) of an LLM, alongside a static, pretrained RoBERTa and a symbolic prior module in two downstream fusions. RDS Fusion demonstrates better performance than an unrestricted reasoning ablation, while requiring substantially lower inference time. We also find that in our experiments, unrestricted reasoning does not yield better results than restricted reasoning. Evaluated on a strictly held-out TweetEval test set (N=734), RDS Fusion achieves 78.1% accuracy and a Macro F1 of 0.777, which is comparable to a finetuned BERTweet-base. On the heavily imbalanced iSarcasm dataset, the frozen CoT pipeline suppresses 22.5% of baseline false positives, yielding a Macro F1 of 0.6726 and Ironic F1 of 0.4821, outperforming multiple heavily supervised SemEval transformer ensembles. Paired McNemar’s tests show that, while adding the symbolic prior to the neural baseline yields an insignificant gain, and the RDS Fusion is statistically insignificant compared to the combined RoBERTa and symbolic prior ablation; the concurrent fusion achieves a statistically significant improvement over the standalone baseline (p=0.005).

In NLP, mental health conditions are often modeled as isolated phenomena, without interpersonal context. We use Reddit posts about long-distance relationships to capture both mental health distress and associated relational triggers. We introduce the Relational Stress and Psychiatry Corpus (RSPC), containing 1,799 Reddit posts annotated by psychiatrists for diagnostic categories, including the most prevalent mood disorders (anxiety and depression), relational stressor triggers, and indications of relationship phase. We benchmark seven fine-tuned transformer models and five large language models across the tasks of multi-label disorder classification, relational trigger detection, and temporal phase prediction. We find clear task-dependent differences between model families, with Claude-3-Haiku achieving the best disorder classification performance (Macro-F1 = 0.538) and GPT-4o obtaining the strongest relational trigger detection performance (Macro-F1 = 0.519), suggesting distinct model capabilities. We further find strong associations between anxiety disorders and chronic relational uncertainty. Overall, RSPC establishes a benchmark for NLP tasks that consider relational context and supports a shift from individual-centric to context-aware mental health modeling, capturing the social and temporal dynamics of distress.

We present Spoiler Court, a multi-agent adversarial framework for interpretable spoiler detection in movie reviews. Three LLM agents, acting as the Plaintiff, Defendant, and Judge, engage in a multi-round courtroom debate to identify spoiler spans, challenge them, and deliver a final judgement with an interpretable audit trail grounded in the movie’s plot summary. Unlike prior work that produces only binary labels, Spoiler Court outputs which sentences are spoilers, why (via plot fact citations), and how the decision was reached (via adversarial debate transcripts). To evaluate the effectiveness of our approach, we construct a multi-genre evaluation dataset spanning 4 movies with binary and character-level span annotations. Under controlled conditions, Spoiler Court achieves an average F1 of 0.733 across 4 movies, outperforming other baselines including a dedicated spoiler detection framework.

Lexical normalization rewrites the noisy, nonstandard words that fill user-generated text (tmrw, u, gr8) into their standard forms. Because labelled data is scarce for most languages, a popular shortcut is to train a single model on many languages at once. We ask a simple question: how many languages should such a model be trained on? Using one fixed-capacity character-level model and twelve languages from a standard benchmark, we vary the number of jointly trained languages from one to twelve and measure per-language accuracy. We find a clear curse of multilinguality: accuracy is highest when a language is trained with only a few others, often just one to four, and then falls steadily and substantially, dropping by about forty percent as the rest are piled on. A control that holds the total amount of training data constant makes the decline arrive sooner and fall further, which points to competition among the languages for one fixed-size model rather than to how much data is available. We also test whether a language’s typological distance from the others predicts its ideal number of cotraining languages, and find no dependable rule: any apparent relationship rests on a couple of languages and does not hold up. For compact normalization models, less can be more: a few languages beat pooling everything into a single model.

In contact centers, real-time agent-assist tools determine, for each of many predefined topics, whether a live customer utterance is relevant and display a coaching card to the agent when it is. The input is noisy and challenging: ASR (Automatic Speech Recognition) transcripts of spontaneous phone conversations, which can be unclear, repetitive, and mostly lack punctuation. To systematically study this real-world task, we curate a human-annotated topic-utterance judgments dataset sourced from real call-center transcripts. We compare three types of matchers: a regex-based baseline, zero-shot sentence-embedding encoders, and Gemini-based LLM matchers. In addition, two types of topic representations are studied in our benchmark: keyphrases and natural language description. Our empirical experiments highlight the superior performance of lightweight LLM matchers when equipped with natural language descriptions.

Natural language processing systems underperform on code-mixed text, particularly for low-resource language pairs. Turkish-English poses a further challenge: it lets English stems combine with Turkish suffixes to form single mixed-language tokens. We introduce TurEngMix, a corpus of 5.5K noisy, naturally occurring social media posts (486,974 tokens) rich in Turkish-English code-mixing. From this corpus, we construct a new Turkish-English benchmark for code-mixed language identification (LID) and named entity recognition (NER), comprising 15K expert-annotated tokens. Evaluating both decoder LLM and fine-tuned encoder baselines, we find that monolingual Turkish and English tokens are labeled reliably, but all models have high error rates on mixed-language tokens for both LID and NER. For morphologically integrated tokens, NER error rates were 5.2× and 6.3× higher for GPT-4o and Qwen, respectively. This highlights how morphological integration remains a challenge. We release the corpus, annotations, and code to support future computational and sociolinguistic research on Turkish-English code-mixing.

Conversational banking assistants on messaging platforms face language that their natural-language-understanding (NLU) pipelines were never designed for: vowel-dropped shorthand (blnc chk krdo), romanized code-mixing, colloquial verb frames. Layered pipelines stack repair rules, learned retrieval, and an LLM, but which layer carries the noise robustness is rarely measured. We conduct targeted interventions on three candidate robustness-bearing components (the normalizer, rules engine, and conditional LLM arbiter) within a fixed 13-layer Hinglish banking NLU (163 intents), with replacement and transfer probes, on BankStress-330, a released three-tier diagnostic stress set, with at least three repeats per LLM-bearing BankStress-330 configuration. LLM-in-the-loop configurations flip up to approximately 3% of verdicts across identical temperature-0 runs, defeating single-run ablation. A 2×2 factorial shows the Hinglish normalizer helps a base encoder but is redundant after noisy-query fine-tuning; the post-fine-tuning redundancy also holds on public Hinglish-TOP (n=6,513, McNemar z=5.0, p≈5×10⁻⁷). Only removing the conditional LLM arbiter collapses adversarial accuracy (.739→.545); neither tested substitute matched it, yet the same fine-tuned cross-encoder improves both noisy tiers while preserving clean accuracy when restricted to reranking (directional, unresolved at n=330). On native-reviewed Tamil and Telugu probes, normalization is unstable in sign. Robustness appears to migrate into tuned components, only for the pair tuned on.