Kelsey R Allen
Author directory2026
ClawBench: Can AI Agents Complete Everyday Online Tasks?
Yuxuan Zhang | Yubo Wang | Yipeng Zhu | Penghui Du | Junwen Miao | Xuan Lu | Zhuofeng Li | Xingwei Qu | Zhengkang Guo | Yuanzhe Shen | Dingjie Song | Han Zhou | Tuney Zheng | Xian Wu | Hao Yu | Songcheng Cai | Yi Lu | Yunzhuo Hao | Minyi Lei | Liang Chen | Kai Zou | Huifeng Yin | Wendong XU | Dongfu Jiang | Ping Nie | Jiaheng Liu | Wenhu Chen | Kelsey R Allen
Findings of the Association for Computational Linguistics: EMNLP 2026
Yuxuan Zhang | Yubo Wang | Yipeng Zhu | Penghui Du | Junwen Miao | Xuan Lu | Zhuofeng Li | Xingwei Qu | Zhengkang Guo | Yuanzhe Shen | Dingjie Song | Han Zhou | Tuney Zheng | Xian Wu | Hao Yu | Songcheng Cai | Yi Lu | Yunzhuo Hao | Minyi Lei | Liang Chen | Kai Zou | Huifeng Yin | Wendong XU | Dongfu Jiang | Ping Nie | Jiaheng Liu | Wenhu Chen | Kelsey R Allen
Findings of the Association for Computational Linguistics: EMNLP 2026
AI agents may assist with emails and documents, but can they reliably complete everyday online workflows on real websites? Everyday online tasks offer a realistic yet unsolved testbed for evaluating next-generation AI agents. To this end, we introduce ClawBench, an evaluation framework comprising 153 everyday online tasks common in daily life and work, spanning 144 platforms across 15 categories, from completing purchases and booking appointments to submitting job applications. These tasks require capabilities beyond existing benchmarks, such as obtaining relevant information from user-provided documents, navigating multi-step workflows across diverse platforms, and write-heavy operations like filling in many detailed forms correctly. Unlike existing benchmarks that evaluate agents in offline sandboxes with static pages, ClawBench operates on production websites, preserving the full complexity, dynamic nature, and interaction challenges of real-world web environments. An interception layer captures and blocks the final submission request, ensuring safe evaluation without real-world side effects. Our evaluations of 8 frontier models show that both proprietary and open-source models complete only a small portion of these tasks. For example, Claude Sonnet 4.6 achieves only 33.3%, which exposes gaps in current AI agents. Progress on ClawBench brings us closer to AI agents that can function as general-purpose assistants.
VGI-Bench: Probing Visual Intelligence in Video Generation Models
Xuan He | Cong Wei | Yuhao Cheng | Linrui Ma | Yuxuan Zhang | Zuojun Li | Yuhao Wen | Jize Jiang | Zeyi Liu | Yuren Hao | Songcheng Cai | Keming Wu | Penghui Du | Kai Zou | Rui Yang | Chenkai Sun | Ke Yang | Ping Nie | Kelsey R Allen | Chenglong Wang | Michel Galley | Jianfeng Gao | ChengXiang Zhai
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Xuan He | Cong Wei | Yuhao Cheng | Linrui Ma | Yuxuan Zhang | Zuojun Li | Yuhao Wen | Jize Jiang | Zeyi Liu | Yuren Hao | Songcheng Cai | Keming Wu | Penghui Du | Kai Zou | Rui Yang | Chenkai Sun | Ke Yang | Ping Nie | Kelsey R Allen | Chenglong Wang | Michel Galley | Jianfeng Gao | ChengXiang Zhai
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Recent studies suggest that video generative models can exhibit certain forms of zero-shot visual reasoning through generated frames. Yet reliable evaluation remains challenging: benchmarks should adopt inputs aligned with the visual priors of current video models, require valid evolving processes rather than only plausible final states, and calibrate task difficulty to remain challenging yet partly feasible. To this end, we introduce VGI-Bench, containing 27 tasks and 810 instances, organized by a two-level taxonomy of domains and skill tags for evaluating the visual reasoning capabilities of video generative models. Our evaluations show that current generative systems can solve a subset of visually grounded reasoning tasks, but remain far from reliable, with even the strongest model, Seedance 2.0, achieving only 51.0% under our evaluation criteria. Our findings further reveal the input sensitivity, skill-bounded transfer from synthetic fine-tuning, and limited denoising-time self-correction, where later steps mainly refine early hypotheses rather than correct reasoning errors. We hope VGI-Bench will help stimulate the development of next-generation video generative models. Data and code are available at [our project page](https://hexuan21.github.io/VGI-Bench/).
Search
Fix author
Co-authors
- Songcheng Cai 2
- Penghui Du 2
- Ping Nie 2
- Yuxuan Zhang 2
- Kai Zou 2
- Liang Chen 1
- Wenhu Chen 1
- Yuhao Cheng 1
- Michel Galley 1
- Jianfeng Gao 1
- Zhengkang Guo 1
- Yunzhuo Hao 1
- Yuren Hao 1
- Xuan He 1
- Dongfu Jiang 1
- Jize Jiang 1
- Minyi Lei 1
- Zhuofeng Li 1
- Zuojun Li 1
- Jiaheng Liu 1
- Zeyi Liu 1
- Xuan Lu 1
- Yi Lu 1
- Linrui Ma 1
- Junwen Miao 1
- Xingwei Qu 1
- Yuanzhe Shen 1
- Dingjie Song 1
- Chenkai Sun 1
- Chenglong Wang 1
- Yubo Wang 1
- Cong Wei 1
- Yuhao Wen 1
- Keming Wu 1
- Xian Wu 1
- Wendong XU 1
- Ke Yang 1
- Rui Yang 1
- Huifeng Yin 1
- Hao Yu 1
- ChengXiang Zhai 1
- Tianyu Zheng 1
- Han Zhou 1
- Yipeng Zhu 1