Poli Nemkova
Author directory2026
Decomposing Self-Reflection in LLM Agents: Diagnostic Scaffolds, Uncertainty Labels, and Action Diversity
Poli Nemkova | Haeshitha Indukuri
Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)
Poli Nemkova | Haeshitha Indukuri
Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)
Self-reflective agents typically bundle monitoring language with a downstream control policy, making it difficult to identify why reflection changes behavior. We present a six-condition ablation that separates evidence re-exposure, diagnostic questions, uncertainty labels, differentiated follow-up actions, and diagnosis–action alignment. Our testbed is an LLM forecasting agent evaluated with Llama-3.3-70B on 310 armed-conflict escalation cases, plus supplementary GPT-4o and 12-country transfer probes. Diagnostic questions produce comparable F1 to free reflection (0.296 vs. 0.297), and adding a seven-label taxonomy while holding the action fixed changes F1 by only +0.008. The closest matched comparison for action differentiation is taxonomy-only versus random action assignment (F vs. D), which holds the typed monitor and taxonomy fixed while replacing the shared-action policy with randomly varying substantive actions: F1 rises from 0.304 to 0.338. Type-matched routing reaches 0.379, but we detect no significant advantage over random assignment (C vs. D, p=0.714). This underpowered contrast does not establish equivalence; it prevents assigning the gain specifically to matching. Brier scores remain flat and transfer is unreliable. Thus, richer reflective traces do not establish better agent control. Our results motivate controls matched for information, vocabulary, and actions when attributing gains in reflection pipelines and identify simple action perturbations as a necessary baseline for type-specific policies.
Whose Good, Whose Place? The Moral Geography of Agentic AI for Social Good
Poli Nemkova | Haeshitha Indukuri | Jaedon Charles
Proceedings of the Fifth Workshop on NLP for Positive Impact (NLP4PI)
Poli Nemkova | Haeshitha Indukuri | Jaedon Charles
Proceedings of the Fifth Workshop on NLP for Positive Impact (NLP4PI)
Agentic AI systems are increasingly proposed for social-good domains, often invoking the United Nations Sustainable Development Goals (SDGs). Yet such claims do not necessarily establish accountability to the communities these systems aim to serve. We present a structured survey of 112 papers on agentic AI for social good published between 2015 and 2026. Each paper is coded across eight dimensions using dual-LLM annotation (GPT-4o and Llama-3.3-70B) with stratified human validation on 50 papers.
We identify a moral-geographic asymmetry: contextual underspecification is concentrated where local context matters most. Overall, 82 of 112 papers (73%) specify no geographic context. Papers aligned with health or physical/ecological SDGs name a geography 37–40% of the time, compared with only 13% for institutional and social-policy SDGs, and 0% under strict coding that requires a concrete country, city, region, or population (𝜒2(2)=8.78, p=.012, Cramér’s V=.283; strict: 𝜒2(2)=7.60, p=.022, V=.263). The pattern is robust across seven of eight sensitivity analyses. SDG 16 is both the most-covered goal and the least geographically specified.
We interpret this pattern as consistent with a form of moral abstraction: institutional good is framed more universally despite its dependence on local political, legal, and cultural conditions. Only 28 papers (25%) report any real-world deployment or small-scale test. We detect no statistically significant differences in deployment rates across architecture, era, or geographic specificity. We conclude with five accountability gaps and a minimal reporting standard for more context-specific, participatory, and accountable research.
We identify a moral-geographic asymmetry: contextual underspecification is concentrated where local context matters most. Overall, 82 of 112 papers (73%) specify no geographic context. Papers aligned with health or physical/ecological SDGs name a geography 37–40% of the time, compared with only 13% for institutional and social-policy SDGs, and 0% under strict coding that requires a concrete country, city, region, or population (𝜒2(2)=8.78, p=.012, Cramér’s V=.283; strict: 𝜒2(2)=7.60, p=.022, V=.263). The pattern is robust across seven of eight sensitivity analyses. SDG 16 is both the most-covered goal and the least geographically specified.
We interpret this pattern as consistent with a form of moral abstraction: institutional good is framed more universally despite its dependence on local political, legal, and cultural conditions. Only 28 papers (25%) report any real-world deployment or small-scale test. We detect no statistically significant differences in deployment rates across architecture, era, or geographic specificity. We conclude with five accountability gaps and a minimal reporting standard for more context-specific, participatory, and accountable research.
Towards Automated Situation Awareness: A RAG-Based Framework for Peacebuilding Reports
Poli Nemkova | Suleyman Olcay Polat | Rafid Ishrak Jahan | Sagnik Ray Choudhury | Sun-Joo Lee | Shouryadipta Sarkar | Mark V. Albert
Proceedings of the Fifth Workshop on NLP for Positive Impact (NLP4PI)
Poli Nemkova | Suleyman Olcay Polat | Rafid Ishrak Jahan | Sagnik Ray Choudhury | Sun-Joo Lee | Shouryadipta Sarkar | Mark V. Albert
Proceedings of the Fifth Workshop on NLP for Positive Impact (NLP4PI)
Timely and accurate situation awareness underpins effective decision-making in humanitarian response, conflict monitoring, and peacebuilding. Yet, synthesizing heterogeneous and rapidly evolving information from news, conflict databases, and economic indicators remains labor-intensive and delays critical interventions. We present a Retrieval-Augmented Generation (RAG) system that autonomously produces structured, evidence-backed situation awareness reports by integrating real-time data from GDELT, ACLED, ReliefWeb, and World Bank APIs. To rigorously assess report quality without ground-truth references, we introduce a three-level reference-free evaluation framework combining automated NLP metrics, expert review by United Nations crisis analysts, and scalable LLM-as-a-Judge assessment. In a multi-country study across 15 conflict-prone regions, our system generated coherent, relevant, and actionable reports, with UNDP analysts estimating a ~50% reduction in report preparation time. These findings demonstrate the feasibility of deploying RAG-based systems for peacebuilding and humanitarian operations and provide a reproducible framework for generating and evaluating AI-assisted situational intelligence at scale.
Hate Speech as a Conflict-Period Marker: Evidence from Arabic Telegram in Sudan
Tara Pai | Poli Nemkova
Proceedings of the Fifth Workshop on NLP for Positive Impact (NLP4PI)
Tara Pai | Poli Nemkova
Proceedings of the Fifth Workshop on NLP for Positive Impact (NLP4PI)
We investigate whether hate-speech intensity on public Telegram channels serves as a conflict signal, using 1,514 days of Arabic-language posts from Sudan aligned with ACLED conflict-event records. Applying a pretrained multilingual hate-speech classifier, we construct a daily intensity index and analyze its relationship to political violence using within-regime analysis and autocorrelation-robust moving-block bootstrap inference. Hate-speech intensity increases approximately fourfold over the months following war onset and remains elevated, but day-to-day fluctuations do not predict short-term changes in violence. Within the post-war period, the index is positively associated with explosions and remote violence (r=+0.37) and negatively associated with protests (r=-0.25) and riots (r=-0.30), with all three associations surviving multiple-comparison correction. These patterns also hold for continuous classifier-score measures, although the regime-level increase is closely intertwined with growth in posting volume and active-channel participation. We further show that pooling across conflict regimes inflates effect sizes, while naive inference on serially dependent time series overstates statistical significance. Hate-speech intensity should therefore be interpreted as a descriptive marker of the conflict-related Telegram environment rather than as an early-warning predictor of imminent violence.
Cross-Lingual Stability and Bias in Instruction-Tuned Language Models for Humanitarian NLP
Poli Nemkova | Amrit Adhikari | Matthew Pearson | Vamsi Krishna Sadu | Albert V. Mark
Proceedings of the Fifteenth Language Resources and Evaluation Conference
Poli Nemkova | Amrit Adhikari | Matthew Pearson | Vamsi Krishna Sadu | Albert V. Mark
Proceedings of the Fifteenth Language Resources and Evaluation Conference
Humanitarian organizations face a critical choice: invest in costly commercial APIs or rely on free open-weight models for multilingual human rights monitoring. While commercial systems offer reliability, open-weight alternatives lack empirical validation - especially for low-resource languages common in conflict zones. This paper presents the first systematic comparison of commercial and open-weight large language models (LLMs) for human-rights-violation detection across seven languages, quantifying the cost-reliability trade-off facing resource-constrained organizations. Across 78,000 multilingual inferences, we evaluate six models - four instruction-aligned (Claude-Sonnet-4, DeepSeek-V3, Gemini-Flash-2.0, GPT-4.1-mini) and two open-weight (LLaMA-3-8B, Mistral-7B) - using both standard classification metrics and new measures of cross-lingual reliability: Calibration Deviation (CD), Decision Bias (ΔBias), Language Robustness Score (LRS), and Language Stability Score (LSS). Results show that alignment, not scale, determines stability: aligned models maintain near-invariant accuracy and balanced calibration across typologically distant and low-resource languages (e.g., Lingala, Burmese), while open-weight models exhibit significant prompt-language sensitivity and calibration drift. These findings demonstrate that multilingual alignment enables language-agnostic reasoning and provide practical guidance for humanitarian organizations balancing budget constraints with reliability in multilingual deployment.
NLP for Social Good: A Survey and Outlook of Challenges, Opportunities and Responsible Deployment
Antonia Karamolegkou | Angana Borah | Eunjung Cho | Sagnik Ray Choudhury | Martina Galletti | Pranav Gupta | Oana Ignat | Priyanka Kargupta | Neema Kotonya | Hemank Lamba | Sun-Joo Lee | Arushi Mangla | Ishani Mondal | Fatima Zahra Moudakir | Deniz Nazar | Poli Nemkova | Dina Pisarevskaya | Naquee Rizwan | Nazanin Sabri | Keenan Samway | Dominik Stammbach | Anna Steinberg Schulten | David Tomás | Steven R Wilson | Bowen Yi | Jessica H Zhu | Arkaitz Zubiaga | Anders Søgaard | Alexander Fraser | Zhijing Jin | Rada Mihalcea | Joel R. Tetreault | Daryna Dementieva
Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Antonia Karamolegkou | Angana Borah | Eunjung Cho | Sagnik Ray Choudhury | Martina Galletti | Pranav Gupta | Oana Ignat | Priyanka Kargupta | Neema Kotonya | Hemank Lamba | Sun-Joo Lee | Arushi Mangla | Ishani Mondal | Fatima Zahra Moudakir | Deniz Nazar | Poli Nemkova | Dina Pisarevskaya | Naquee Rizwan | Nazanin Sabri | Keenan Samway | Dominik Stammbach | Anna Steinberg Schulten | David Tomás | Steven R Wilson | Bowen Yi | Jessica H Zhu | Arkaitz Zubiaga | Anders Søgaard | Alexander Fraser | Zhijing Jin | Rada Mihalcea | Joel R. Tetreault | Daryna Dementieva
Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Natural language processing (NLP) now shapes many aspects of our world, yet its potential for positive social impact is underexplored. This paper surveys work in “NLP for Social Good" (NLP4SG) across nine domains relevant to global development and risk agendas, summarizing principal tasks and challenges. We analyze ACL Anthology trends, finding that inclusion and AI harms attract the most research, while domains such as poverty, peacebuilding, and environmental protection remain underexplored. Guided by our review, we outline opportunities for responsible and equitable NLP and conclude with a call for cross-disciplinary partnerships and human-centered approaches to ensure that future NLP technologies advance the public good.
Search
Fix author
Co-authors
- Sagnik Ray Choudhury 2
- Haeshitha Indukuri 2
- Sun-Joo Lee 2
- Amrit Adhikari 1
- Mark V. Albert 1
- Angana Borah 1
- Jaedon Charles 1
- Eunjung Cho 1
- Daryna Dementieva 1
- Alexander Fraser 1
- Martina Galletti 1
- Pranav Gupta 1
- Oana Ignat 1
- Rafid Ishrak Jahan 1
- Zhijing Jin 1
- Antonia Karamolegkou 1
- Priyanka Kargupta 1
- Neema Kotonya 1
- Hemank Lamba 1
- Arushi Mangla 1
- Albert V. Mark 1
- Rada Mihalcea 1
- Ishani Mondal 1
- Fatima Zahra Moudakir 1
- Deniz Nazar 1
- Tara Pai 1
- Matthew Pearson 1
- Dina Pisarevskaya 1
- Suleyman Olcay Polat 1
- Naquee Rizwan 1
- Nazanin Sabri 1
- Vamsi Krishna Sadu 1
- Keenan Samway 1
- Shouryadipta Sarkar 1
- Anna Steinberg Schulten 1
- Dominik Stammbach 1
- Anders Søgaard 1
- Joel Tetreault 1
- David Tomás 1
- Steven R Wilson 1
- Bowen Yi 1
- Jessica H Zhu 1
- Arkaitz Zubiaga 1