Output-token budgets are a first-order deployment constraint for large language model (LLM) agents, yet existing benchmarks typically evaluate them under a single fixed or generous budget. We introduce TokenCliff, a benchmark for measuring multi-turn agent performance across episode-level output-token budgets. It contains 660 deterministic tasks spanning four capability-oriented domains—file operations, data transformation, tool use, and planning—plus a compositional multi-step split. We propose two budget-sensitivity metrics: Budget Elasticity, which captures robustness across budget levels, and the Cliff Index, which captures the sharpest adjacent-budget performance drop. Evaluating 10 LLMs at five budget levels from 200 to 8,192 tokens yields 33,000 episodes. Results show that (1) accuracy drops by up to 32 percentage points between adjacent budgets; (2) neither peak performance nor larger model size necessarily implies stronger budget elasticity; and (3) budget sensitivity is substantially greater in the evaluated multi-turn setting than in the single-shot diagnostic. Code and data are available in our public repository: https://github.com/BairuiZhang/TokenCliff

Social deduction games like Werewolf combine language, reasoning, and strategy, providing a testbed for studying language and social intelligence. However, most studies reduce the game to LLM-based self-play, yielding templated utterances and anecdotal cases that overlook the richness of social gameplay. Evaluation further relies on coarse metrics such as survival time or subjective scoring due to the lack of quality reference data. To address these gaps, we curate a high-quality, human-verified, multilingual, and multimodal Werewolf dataset containing over 100 hours of video, 32.4M utterance tokens, and 15 rule variants. Based on this dataset, we propose a novel strategy-alignment evaluation that leverages the winning faction’s strategies as ground truth in two stages: 1) Speech evaluation, formulated as multiple-choice-style tasks that assess whether the model can adopt appropriate stances across five dimensions of social ability. 2) Decision evaluation, which assesses the model’s voting choices and opponent-role inferences. This framework enables a fine-grained evaluation of models’ linguistic and reasoning capabilities, while capturing their ability to generate strategically coherent gameplay. Our experiments show that state-of-the-art LLMs and VLMs exhibit diverse performance, with roughly half of the models in both modalities remaining below 0.50 accuracy, revealing clear gaps in deception and counterfactual reasoning.

Multi-vector visual document retrieval models encode pages into hundreds of patch-level embeddings, enabling fine-grained interaction but incurring substantial storage overhead. Existing train-free compression approaches rely on token pruning or heuristic merging, which often fail to preserve the broad semantic coverage of the original representations under constrained budgets. To address this, we propose CovMerge, a geometry-aware, train-free index compression framework that formulates token reduction as a budgeted semantic coverage problem on the unit hypersphere. Given frozen patch embeddings, CovMerge applies Farthest Point Sampling (FPS) for coverage-aware prototype initialization, followed by spherical K-Means refinement aligned with MaxSim retrieval. The primary variant, Global Coverage, operates directly on all page patches without external dependencies, providing the strongest simplicity–efficiency trade-off. An optional Regional Coverage extension leverages document layout structure to guide prototype allocation, offering additional gains on layout-sensitive documents when reliable detection is available. Operating purely as post-processing without retraining, CovMerge supports flexible deployment budgets. On the ViDoRe V1 benchmark, CovMerge retains up to 98.1% and 98.6% of the original nDCG@5 for ColPali and ColQwen2, respectively, while using only 10% of the original storage, establishing a new state-of-the-art for storage-efficient multimodal retrieval.