Ayush Nangia
Author directory2026
UNSC-Bench: Evaluating LLM Diplomatic Role-Playing Through UN Security Council Vote Prediction
Ayush Nangia | Aman Gokrani | Ruggero Marino Lazzaroni
Proceedings of the First Workshop on Multilingual Multicultural Evaluation
Ayush Nangia | Aman Gokrani | Ruggero Marino Lazzaroni
Proceedings of the First Workshop on Multilingual Multicultural Evaluation
This paper introduces UNSC-Bench, a benchmark for evaluating Large Language Models (LLMs) in simulating diplomatic decision-making through United Nations Security Council (UNSC) vote prediction. The dataset includes 469 UNSC resolutions from 1947 to 2025, with voting records for the five permanent members (P5) (United States, China, France, Russia, United Kingdom) and translations in four languages. We analyze 26 LLMs, along with thinking variants, across multiple P5 roles and find that (1) without explicit role assignment, models are diplomatically unaligned, defaulting to high yes rates and failing to match any P5 voting pattern, indicating they lack inherent diplomatic identity; (2) model capability (as measured by MMLU-Pro) is strongly correlated with role-playing accuracy; (3) regional models do not outperform others in predicting their home country’s votes; and (4) multilingual evaluation reveals that prompt language impacts model predictions, particularly for minority vote outcomes.
Do Tools Matter, or Just Their Descriptions?
Ananya Tomar | Ayush Nangia
Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing
Ananya Tomar | Ayush Nangia
Proceedings of the 11th Workshop on Financial Technology and Natural Language Processing
Tool-augmented language models are increasingly applied to financial document reasoning, a setting that demands numerical precision over long financial filings. Earlier comparisons either mix up the model with its scaffolding or report tool-use gains without separating code, strategy, and environment. We isolate these factors in a controlled same-model ablation: six configurations share an identical GPT-5 backbone and differ only in the tool surface (callable tools plus their descriptions), with one control that additionally forces sub-model delegation. On DocMath-Eval under oracle context, the fully equipped configuration beats the restricted-tool baseline by +3.6 points (p=0.0028). Where this gain comes from matters more than its size: the system tools that give that configuration its name are used on under 0.2% of samples, and a description-only control, which describes the same tools but never runs them, performs almost identically (within 1 point on every split). The benefit comes from the tool descriptions, not from executing the tools. On long-document splits, the equipped model uses fewer iterations, has answers available earlier when runs are cut short, and varies its tool mix with document length. Comprehension errors dominate the hardest failures: the bottleneck is document comprehension, not the tool interface.