Large Language Models (LLMs) have demonstrated remarkable capabilities in processing long contexts and generating long-form texts. However, existing benchmarks evaluate these capabilities by focusing on either long-input comprehension with short outputs or short-input creative generation with lengthy outputs. This critical gap overlooks real-world Long-In-Long-Out Generation tasks, where models must simultaneously process long-context inputs and produce long-form outputs while adhering to fine-grained constraints. We introduce DocFollow, a comprehensive benchmark featuring 2,436 expert-validated instances across 5 major domains and 20 subdomains, designed to evaluate LLMs’ ability to follow hierarchical constraints in Document Following tasks. Our benchmark employs five types of constraints with increasing granularity. To ensure reliable evaluation, we develop a rule-based assessment protocol that decomposes overall evaluation uncertainty into atomic-level classification uncertainties, employing LLMs solely as information extractors for well-established NER and RE tasks rather than subjective scoring. Our evaluation of 13 state-of-the-art LLMs reveals a consistent pattern: while models maintain reasonable performance on coarse-grained constraints, they exhibit substantial degradation on fine-grained requirements, particularly entity-level constraints. This finding suggests fundamental architectural limitations in maintaining detailed structural and semantic fidelity throughout extended generation. Our findings highlight the critical need for advancing long-context generation capabilities beyond surface-level coherence to achieve precise fine-grained control. The implementation is publicly available. Source code is available at https://github.com/leonbig/DocFollow

Despite recent progress, existing agent benchmarks neglect a fundamental real-world capability: hierarchical rule application, a critical requirement in fields such as law and medicine where agents must reason from broad categories down to specific exceptions to reach rule-compliant decisions.This introduces significant challenges in resolving logical dependencies and disambiguating vague boundaries.To bridge this gap, we introduce HSCodeComp, a novel benchmark derived from e-commerce, requiring agents to assign a unique 10-digit Harmonized System (HS) Code to products by aligning their fuzzy attributes with strict tariff classification rules.HSCodeComp comprises 632 realistic products across 32 categories, featuring detailed yet noisy product information (titles, attributes, and images). The HS Codes are annotated by a panel of 26 tariff experts, strictly adhering to official rules and an empirical knowledge base, both of which we jointly open-source.Through a comprehensive evaluation of 23 LLMs, VLMs, and agents on HSCodeComp, we demonstrate that: 1) a substantial performance gap remains between state-of-the-art agents and human experts (46.8% vs. 95.0%); and 2) test-time scaling fails to close this gap. Further analysis reveals that 1) excessive reasoning steps frequently induce “reasoning drift,” which degrades accuracy; and 2) agents are prone to premature decisions on high-level categories and reasoning hallucinations that lack factual grounding.

We present a statistical framework for modeling and controlling large language model (LLM) response lengths using extreme value theory. Analyzing 14,301 GPT-4o responses across temperature and prompting conditions, with cross-validation on Qwen and DeepSeek architectures, we demonstrate that verbosity follows Weibull-type generalized extreme value (GEV) distributions with heavier tails under stochastic generation. Our key contributions include: (1) development of a novel GEV-generalized Pareto (GPD) hybrid model that improves tail fit (R²_CDF=0.9993 vs standalone GEV’s 0.998) while maintaining architectural generalizability; (2) quantitative characterization of prompt anchoring effects across models, showing reduced dispersion but increased outliers under randomization; and (3) identification of temperature-dependent response patterns that persist across architectures, with higher temperatures amplifying length variability while preserving extreme-value mechanisms. The hybrid model’s threshold selection method enables precise verbosity control in production systems regardless of model choice. While validated on multiple architectures, generalizability to emerging model families requires further study.