Vision Large Language Models (VLLMs) show promise for multi-modal understanding; yet, their application to emotion recognition remains limited by insufficient spatial and contextual awareness and by supervised fine-tuning that uniformly penalizes all reasoning tokens—suppressing valid alternatives. We present Set-of-Vision-Text Rewards (SoVTR), a unified, reward-based framework that guides VLLMs with interpretable and verifiable visual attributes, focusing optimization on the emotion-discriminative reasoning steps that truly determine correctness. SoVTR decomposes supervision into three rewards: (i) Face Reward capturing action units, gaze, and micro-expressions; (ii) Body Reward encoding pose and gesture cues; and (iii) Context Reward modeling person–object relations and scene activities. By combining BERTScore attribute matching with GRPO policy optimization, SoVTR produces attribute-grounded explanation traces while preserving non-critical reasoning diversity. Across DFEW, FERV39k, and ExpW, SoVTR consistently improves both recognition accuracy and grounding quality, delivering substantial gains over strong VLLM baselines.