Siyuan Liu

Author directory

Other people with similar names: Siyuan Liu, Siyuan Liu


2026

We introduce EMemBench, a programmatic benchmark generator for evaluating long-term episodic memory of agents through interactive games. Rather than using a fixed set of questions, EMemBench generates questions from environment-grounded trajectories, covering both text-only and visual game environments. Each template computes verifiable ground truth from underlying game signals, with controlled answerability and balanced coverage over memory skills: single/multi-hop recall, induction, temporal, spatial, logical, and adversarial. We evaluate memory agents with strong LMs/VLMs as backbones, using in-context prompting as baselines. Across 15 text games and multiple visual seeds, results are far from saturated: induction and spatial reasoning are persistent bottlenecks, especially in visual settings. Persistent memory yields clear gains for open backbones on text games, but improvements are less consistent for VLM agents, suggesting that visually grounded episodic memory remains an open challenge. A human study further contextualizes the difficulty and interpretability of EMemBench.
Skills are increasingly used to improve LLM agents on real-world tasks, yet it remains unclear how agents use skills, how skill quality should be evaluated, and how users should choose skills under cost-performance trade-offs. We present OpenSkillEval, an automatic framework for evaluating both skill-augmented agents and skills. OpenSkillEval dynamically constructs test cases from evolving real-world artifacts across five applications: presentation generation, front-end web design, poster generation, data visualization, and report generation. It also curates community skills for controlled comparison under unified settings. Using over 600 generated tasks and 30 open-source skills, we evaluate state-of-the-art models and agent frameworks. Results show that skill availability does not guarantee effective use, skill benefits depend strongly on the model and framework, and many popular skills fail to consistently outperform skill-free agents. These findings highlight the need for dynamic, task-grounded evaluation and offer practical guidance for skill design, selection, and deployment.