Renato Okabayashi Miyaji
Author directory2026
Rethinking Aleatoric Uncertainty: Partitioning Irreducible States for Large Language Model Agents
Renato Okabayashi Miyaji | Matias Valdenegro-Toro
The Proceedings of the Third Workshop on Uncertainty-Aware NLP
Renato Okabayashi Miyaji | Matias Valdenegro-Toro
The Proceedings of the Third Workshop on Uncertainty-Aware NLP
While Uncertainty Quantification (UQ) remains an active research field for Large Language Models (LLMs), the current literature lacks a rigorous understanding of what constitutes aleatoric uncertainty and its exact sources. Existing methodologies largely rely on the traditional dichotomy of aleatoric and epistemic uncertainty, which inadequately treats uncertainty as a static snapshot. To bridge this gap, this position paper challenges the static treatment of uncertainty. Supported by empirical evidence, we posit that aleatoric uncertainty in LLM agents is fundamentally dynamic and must be partitioned into strictly and temporarily irreducible states—differentiating intrinsic task variance, such as multiple valid answers, from resolvable noise, including input ambiguity, typographical errors, and contextual conflict. Through episodic empirical evaluations across multi-turn scenarios, we demonstrate how targeted agentic interventions can actively resolve temporarily irreducible noise, transitioning the interaction into an epistemic regime. When data uncertainty is strictly irreducible, model interventions cannot isolate a single correct distribution. While we acknowledge that the traditional static dichotomy remains a valuable framework for non-agentic settings, we urge the community to develop new decomposition paradigms that track uncertainty along the timeline of interaction. Embracing this dynamic perspective is essential to achieve robust uncertainty quantification for modern LLM agents.
MACE-XAI: A Multi-Agent Crowd Evaluation Framework for Robust Natural Language Explanations
Renato Okabayashi Miyaji
Proceedings of the 9th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP
Renato Okabayashi Miyaji
Proceedings of the 9th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP
The rapid integration of Machine Learning in critical decision-making has shifted the focus of Explainable Artificial Intelligence (XAI) toward human-centric Natural Language Explanations (NLEs). While Large Language Models (LLMs) can synthesize narrative justifications from technical feature attributions, ensuring the faithfulness and robustness of these explanations remains an open challenge. Although multi-agent ensembles and Crowd-based Comparative Evaluation (CCE) are emerging standards for evaluating open-ended dialogues, they fail to capture domain-specific hallucinations and structural omissions inherent to technical explanations. To bridge this gap, we introduce MACE-XAI, a novel neuro-symbolic framework engineered specifically for the robust assessment of NLEs in the context of tabular feature-attribution XAI. Rather than proposing isolated evaluators, our key innovation lies in the architectural synergy of its components: we orchestrate a heterogeneous crowd of LLM agents to evaluate pragmatic text quality and fluency, while anchoring them with a deterministic symbolic verification layer that audits textual claims against raw technical ground truths. Experimental results demonstrate that while standalone LLM-as-judge setups are easily swayed by linguistic surface features, MACE-XAI achieves an agreement with human experts (Cohen’s 𝜅=0.64). While this represents a significant improvement over standalone neural judges, it also highlights the remaining alignment gap that must be addressed before fully autonomous evaluation in safety-critical deployments.