In post-training large language models (LLMs), supervised fine-tuning (SFT) provides expert guidance but can overfit, while reinforcement learning (RL) enables exploration but often suffers from cold-start and sparse-reward issues. Although combining SFT with RL is intuitive, naively mixing off-policy expert data into on-policy RL can induce distribution shift and destabilize optimization. We propose APEX, a framework that harmonizes off-policy guidance and on-policy exploration via a Cognitive State Awareness mechanism. Specifically, APEX leverages a perplexity-based Student’s -test to compare the sequence-level perplexity distributions of on-policy generations against candidate offline data. This comparison diagnoses whether failures reflect sampling stochasticity or missing knowledge. Guided by this diagnosis, APEX adaptively gates GRPO optimization and modulates SFT supervision. The framework preferentially selects offline samples that are statistically consistent with the current on-policy distribution to guide RL updates, while filtering out high-shift data and verifiably incorrect samples that could destabilize training. APEX also uses a dynamically weighted SFT loss when needed. Extensive experiments show that APEX improves over SFT-then-RL by +7.10 points in average pass@1 on six math benchmarks and by +15.50 points on three general reasoning benchmarks.

Group Relative Policy Optimization (GRPO) is one of the most widely adopted RLVR algorithms for post-training large language models on reasoning tasks. We first show that GRPO admits an equivalent discriminative reformulation, in which policy optimization maximizes the expected score gap between verified positive and negative rollouts. This reformulation reveals two objective-level limitations: likelihood-misaligned surrogate scores, in which clipped ratio-based scores are optimized rather than the sequence likelihoods that govern generation, and score-insensitive credit assignment, in which rollout-level credit does not reflect the current score gaps between positive and negative rollouts. To address these limitations, we propose ConSPO, a Contrastive Sequence-level Policy Optimization method that uses length-normalized sequence log-probabilities as rollout scores and contrasts verified positive rollouts against negative distractors within the same group. ConSPO optimizes a group-wise InfoNCE-style objective to adaptively strengthen updates for poorly separated positives and high-scoring negatives, together with a curriculum-scheduled margin that preserves separation pressure as training progresses. Experiments across diverse settings show that ConSPO outperforms strong baselines on challenging reasoning benchmarks.

Large multimodal models (LMMs) inherit the self-attention mechanism of pretrained language backbones, yet standard attention can exhibit suboptimal allocation, including cross-modal misallocation between textual and visual evidence and intra-visual imbalance among visual tokens. We propose RAVE (Re-Allocating Visual Attention), a lightweight pair-gating mechanism that adds a learned query–key bias to pre-softmax attention scores over visual keys, derived from pre-RoPE query and key features. RAVE requires no architectural modification to the backbone and can be trained end-to-end with the rest of the model. Across a suite of multimodal benchmarks, RAVE improves over standard attention by 3 points on average, with the largest gains on perception-intensive tasks—including multilingual OCR, chart understanding, document VQA, and scene text VQA—where accurate visual grounding is critical.