Vision Large Language Models (VLLMs) show promise for multi-modal understanding; yet, their application to emotion recognition remains limited by insufficient spatial and contextual awareness and by supervised fine-tuning that uniformly penalizes all reasoning tokens—suppressing valid alternatives. We present Set-of-Vision-Text Rewards (SoVTR), a unified, reward-based framework that guides VLLMs with interpretable and verifiable visual attributes, focusing optimization on the emotion-discriminative reasoning steps that truly determine correctness. SoVTR decomposes supervision into three rewards: (i) Face Reward capturing action units, gaze, and micro-expressions; (ii) Body Reward encoding pose and gesture cues; and (iii) Context Reward modeling person–object relations and scene activities. By combining BERTScore attribute matching with GRPO policy optimization, SoVTR produces attribute-grounded explanation traces while preserving non-critical reasoning diversity. Across DFEW, FERV39k, and ExpW, SoVTR consistently improves both recognition accuracy and grounding quality, delivering substantial gains over strong VLLM baselines.

Vision Large Language Models (VLLMs) are transforming the intersection of computer vision and natural language processing; however, the potential of using visual prompts for emotion recognition in these models remains largely unexplored and untapped. Traditional methods in VLLMs struggle with spatial localization and often discard valuable global context. We propose a novel Set-of-Vision prompting (SoV) approach that enhances zero-shot emotion recognition by using spatial information, such as bounding boxes and facial landmarks, to mark targets precisely. SoV improves accuracy in face count and emotion categorization while preserving the enriched image context. Through comprehensive experimentation and analysis of recent commercial or open-source VLLMs, we evaluate the SoV model’s ability to comprehend facial expressions in natural environments. Our findings demonstrate the effectiveness of integrating spatial visual prompts into VLLMs for improving emotion recognition performance.