Yooju Shin

Author directory

2026

Unified Multi-Modal Generation (UMMG) models support seamless switching between generation tasks spanning multiple modality combinations within a single conversational interface. However, existing safety benchmarks construct prompts independently per modality, so different modalities cover different harmful intentions, preventing holistic evaluation of UMMG models. We propose an intention-grounded evaluation methodology in which every modality combination is evaluated against the same set of harmful intentions, yielding fair, directly comparable safety scores across modalities. Because scores are aggregated at the intention level — averaging over the same set of intentions rather than over modality-specific prompt sets — each model receives an accurate safety profile not distorted by per-modality differences. The methodology is incrementally extensible: new modality combinations or intentions can be added without rebuilding existing data. Following this methodology, we construct UNISAF-Bench (UNIfied intention-grounded SAFety Benchmark): 1,257 human-verified harmful intentions across 19 policies, yielding 20,112 conversations spanning 4 modality combinations, single-/multi-turn settings, and paired policy-violating/boundary-benign variants. Evaluation of four open-source UMMG models reveals systematic safety gaps: multimodal inputs degrade text-output safety, safety alignment fails to transfer to image generation, and multi-turn deflection bias inflates refusal on both harmful and benign prompts.
Bargaining is often regarded as a logical arena rather than an art or a matter of intuition, yet Large Language Models (LLMs) still struggle to navigate it due to limited strategic depth and difficulty adapting to complex human factors. Current benchmarks rarely capture this limitation. To bridge this gap, we present a utility feedback centric framework. Our contributions are: (i) AgoraBench, a new benchmark spanning nine challenging settings (e.g., deception, monopoly) that supports diverse strategy modeling; (ii) human-aligned, economically grounded metrics derived from utility theory. This is operationalized via agent utility, negotiation power, and acquisition ratio that implicitly measure how well the negotiation aligns with human preference and (iii) a human preference grounded dataset with learning pipeline that strengthens LLMs’ bargaining ability through both prompting and finetuning. Empirical results indicate that baseline LLM strategies often diverge from human preferences, while our mechanism substantially improves negotiation performance, yielding deeper strategic behavior and stronger opponent awareness.