To achieve complex real-world goals, AI agents must be able to plan appropriate sequences of actions from a task specification. However, evaluating and improving the planning capabilities is challenging because planning involves multiple steps that are difficult to disentangle when viewed as a whole. A more feasible approach is to focus on Temporal Execution Order (TEO), a core planning sub-task that requires partially ordering actions based on dependency by reasoning about temporal and causal dependency constraints. Existing research studying models’ reasoning over temporal execution is limited to automatically derived annotations, approximations of TEO as a linear chain, or text-only inputs. To address this gap, we introduce MATEO (MultimodAl Temporal Execution Order), a multimodal benchmark designed to assess and improve the temporal reasoning abilities of Large Vision Language Models (LVLMs). We acquire a high-quality multimodal recipe corpus, authored through a standardized editorial process that decomposes instructions into discrete steps, each paired with corresponding images. We design a novel set of guidelines and procedures to collect TEO annotations as graphs via crowdsourcing, which can be extended to new recipes, languages, and domains. Using MATEO, we evaluate six state-of-the-art LVLMs across model scales, varying language context, multimodal input structure, and fine-tuning strategies, showing the importance of multimodal inputs as well as limitations of these models.

While Vision-Language Models (VLMs) have achieved competitive performance in various tasks, their comprehension of the underlying structure and semantics of a scene remains understudied. To investigate the understanding of VLMs, we study their capability regarding object properties and relations in a controlled and interpretable manner. To this scope, we introduce CIVET, a novel and extensible framework for systematiC evaluatIon Via controllEd sTimuli. CIVET addresses the lack of standardized systematic evaluation for assessing VLMs’ understanding, enabling researchers to test hypotheses with statistical rigor. With CIVET, we evaluate five state-of-the-art VLMs on exhaustive sets of stimuli, free from annotation noise, dataset-specific biases, and uncontrolled scene complexity. Our findings reveal that 1) current VLMs can accurately recognize only a limited set of basic object properties; 2) their performance heavily depends on the position of the object in the scene; 3) they struggle to understand basic relations among objects. Furthermore, a comparative evaluation with human annotators reveals that VLMs still fall short of achieving human-level accuracy.