Large Language Models (LLMs) excel when a solution can be planned from the prompt, but many sequential decision-making tasks require actions to depend on observations acquired during interaction. We study two central challenges in this regime: learning reusable interaction policies for history-dependent search, and acting under partial observability when task-relevant state is revealed only through interaction. We instantiate these challenges with two diagnostic benchmarks: Calc-Bench for reusable, history-dependent tool interaction, and partially observed sorting for action selection under hidden state. We introduce the Expanded Action space (ExpA), which decouples language reasoning from environment interaction by adding routing and environment-specific actions beyond vocabulary tokens. ExpA is trained with EARL, an RL algorithm using counterfactual policy optimization to explore rarely invoked but useful actions. EARL improves Countdown by 26.3% with 3B models and achieves perfect Sort-4 accuracy while discovering an efficient sorting strategy, providing a proof-of-concept for learned non-language interaction in controlled sequential decision-making diagnostics.

Whether attention weights faithfully reflect model reasoning has been actively debated in NLP, yet this question remains largely unexplored for the visual modality in Vision-Language Models (VLMs). We address this gap through causal perturbation analysis on current VLMs, evaluating both the comprehensiveness and sufficiency gap of attention-ranked visual tokens. Our analysis reveals that visual attention faithfulness is heterogeneous, manifesting in three distinct processing modes: *Faithful-Sufficient*, where top-k attention tokens are both necessary and sufficient for prediction; *Faithful-Distributed*, where they are necessary but broader visual context remains required; and *Non-Focal*, where no localized attention region is individually necessary while visual information remains an essential trigger for prediction. Furthermore, human-annotated ground-truth regions satisfy comprehensiveness in only ∼60% of cases compared with model attention rankings, revealing systematic divergence between model visual reliance and human intuition. We demonstrate these patterns across both general VQA on VQAv2 and document tasks on VRDU and ChartQA, showing that visual attention faithfulness varies systematically with processing demands and model architectures rather than being uniformly faithful or unfaithful.