Recent advances in image and video generation have highlighted the need for intuitive, high-precision manipulation. However, traditional text-based prompts or 2D Visual Prompts (e.g., dragging, 2D trajectories) often suffer from unintuitive prompt engineering or depth ambiguity. To address these limitations, we introduce 3D Visual Prompting, a novel paradigm that significantly improves input intuitiveness for visual manipulation tasks. We develop a framework that integrates intuitive 3D constraints into 2D generative processes. Our approach projects 3D Visual Prompts onto 2D visual planes and leverages Multimodal Large Language Models (MLLMs) to verbalize intent, allowing plug-and-play use of existing 2D models. Furthermore, we incorporate In-Context Self-Refinement (ICSR) to summarize insights from self-correction into a meta-strategy. This enables high-precision 3D control in single-pass inference, eliminating the need for iterative computation. Experimental results demonstrate that our method significantly outperforms traditional 2D Visual Prompting for object translation and rotation. By utilizing pre-trained models without additional parameter tuning, our framework remains highly adaptable to future developments in generative models.

We study the vulnerability of guardrail models to adversarial paraphrases that exploit superficial textual variation: meaning-preserving changes that alter expressions spuriously correlated with safety labels. While prior work probes this vulnerability through simple heuristics, such as keyword-label co-occurrence statistics, we argue that guardrails may rely on more complex textual expressions involving phrases or syntactic structures that cannot be captured by heuristics. To enable systematic evaluation of such hidden vulnerabilities, we propose ASPro, an agentic adversarial evaluation framework that automatically synthesizes meaning-preserving paraphrases containing expressions that contradict guardrails’ learned spurious expression-label correlations. Applying ASPro to five safety benchmarks and 13 guardrail models, we find that the average F1 drops from 77.83 to 40.25, substantially more than under existing heuristic baselines. Moreover, on ASPro prompts, guardrail models often fail to outperform their base LLMs before safety fine-tuning. These results suggest that current safety fine-tuning may overfit to spurious expressions rather than truly improve guardrails’ understanding of prompt safety.

We introduce a voice-agentic framework that learns one critical omni-understanding skill: knowing when to trust itself versus when to consult external audio perception. Our work is motivated by a crucial yet counterintuitive finding: naively fine-tuning an omni-model on both speech recognition and external sound understanding tasks often degrades performance, as the model can be easily misled by noisy hypotheses. To address this, our framework, Speech-Hands, recasts the problem as an explicit self-reflection decision. This learnable reflection primitive proves effective in preventing the model from being derailed by flawed external candidates. We show that this agentic action mechanism generalizes naturally from speech recognition to complex, multiple-choice audio reasoning. Across the OpenASR leaderboard, which includes seven domain-diverse speech datasets, Speech-Hands consistently outperforms strong baselines by 12.1% WER on the OpenASR benchmark. The model also achieves 77.37% accuracy and high F1 on audio QA decisions, showing robust generalization and reliability across diverse audio question answering datasets. By unifying perception and decision-making, our work offers a practical path toward more reliable and resilient audio intelligence.

Large Vision-Language Models (LVLMs) have transformed image captioning, shifting from concise captions to detailed descriptions. We introduce LOTUS, a leaderboard for evaluating detailed captions, addressing three main gaps in existing evaluations: lack of standardized criteria, bias-aware assessments, and user preference considerations. LOTUS comprehensively evaluates various aspects, including caption quality (e.g., alignment, descriptiveness), risks (e.g., hallucination), and societal biases (e.g., gender bias) while enabling preference-oriented evaluations by tailoring criteria to diverse user preferences. Our analysis of recent LVLMs reveals no single model excels across all criteria, while correlations emerge between caption detail and bias risks. Preference-oriented evaluations demonstrate that optimal model selection depends on user priorities.

Large language models (LLMs) have enhanced the capacity of vision-language models to caption visual text. This generative approach to image caption enrichment further makes textual captions more descriptive, improving alignment with the visual context. However, while many studies focus on the benefits of generative caption enrichment (GCE), are there any negative side effects? We compare standard-format captions and recent GCE processes from the perspectives of gender bias and hallucination, showing that enriched captions suffer from increased gender bias and hallucination. Furthermore, models trained on these enriched captions amplify gender bias by an average of 30.9% and increase hallucination by 59.5%. This study serves as a caution against the trend of making captions more descriptive.