Ruggero Marino Lazzaroni

Author directory

2026

This paper introduces UNSC-Bench, a benchmark for evaluating Large Language Models (LLMs) in simulating diplomatic decision-making through United Nations Security Council (UNSC) vote prediction. The dataset includes 469 UNSC resolutions from 1947 to 2025, with voting records for the five permanent members (P5) (United States, China, France, Russia, United Kingdom) and translations in four languages. We analyze 26 LLMs, along with thinking variants, across multiple P5 roles and find that (1) without explicit role assignment, models are diplomatically unaligned, defaulting to high yes rates and failing to match any P5 voting pattern, indicating they lack inherent diplomatic identity; (2) model capability (as measured by MMLU-Pro) is strongly correlated with role-playing accuracy; (3) regional models do not outperform others in predicting their home country’s votes; and (4) multilingual evaluation reveals that prompt language impacts model predictions, particularly for minority vote outcomes.
We describe a system for the LM-KBC 2026 shared task on closed-book knowledge base construction. Given a subject and relation, the system tries to predict all corresponding objects without retrieval or additional training and remains within the 32-billion-parameter limit. The system uses the base model gemma-4-31B with separate prompting and aggregation rules for six relations; few-shot demonstrations and selection thresholds come from the train set and are checked on the validation set. It obtains 0.6975 global F1 on the official validation set, compared with 0.308 for the organiser baseline on validation. The largest per-relation gains over our earlier baseline configuration come from a company-listing prompt with explicit abstention and a log-stratified area prompt, plus a vote-share abstention threshold for empty-heavy relations (like personHasCityOfDeath) that recovers most of the empty-case precision. Across more than 30 prompting, verification, and aggregation configurations, further inference rarely recovers facts that are absent from the generated candidates. These results indicate that relation-specific elicitation improves closed-book extraction, while candidate coverage and selection remain the main limitations. Code, prompts, and configuration files are available at https://github.com/ruggsea/lmkbc2026-submission.

2025