Agentic LLM systems that generate code through multi-turn tool use face a fundamental context problem: each session starts from zero, discarding the domain constraints, data schemas, tool configurations, and output preferences that made previous sessions productive. We introduce shared selective persistent memory, an architecture that retains four categories of reusable context — task specifications, data schemas, tool configurations, and output constraints — while discarding session-specific reasoning traces, and that packages them into workspaces transferable across users under role-based access control.

The resulting cost curve is non-monotonic. In a controlled replication on four public datasets, where a formatting specification is established once and then withheld, no memory completes 0/12 trials at 3.8K input tokens, selective memory completes 12/12 at 3.9K, and full conversation history completes 8/12 at 7.7K. What is kept matters more than how much is kept: the winning configuration costs essentially what the failing one does, and twice as much context does not improve on it. Both differences from no memory survive Bonferroni-corrected exact McNemar tests (p = 0.0005, p = 0.008); selective memory and full history are indistinguishable on completion (p = 0.125) and separate on price. Every no-memory failure is a specification failure and nothing else — the artifacts are structurally sound but do not reproduce a constraint they were never given.

We implement this in a deployed platform where agents produce git-versioned artifacts from CSV, SQL, REST, and MCP sources. A complementary zero-token data refresh contract decouples generated programs from runtime data, firing on 12/12 trials at a median 0.08s with no model call, while summary-driven data representation costs 97–431x fewer tokens than raw injection. Across 24 recurring enterprise tasks selective memory completes 23/24 against 19/24 and 17/24, though at that sample no pairwise difference reaches significance; there, where users re-specify rather than rely on memory, full history returns no improvement at all despite 9x the input.

Modern edge AI systems rely on dynamic, multi-stage pipelines vulnerable to adversarial attacks. Existing attacks primarily target static tasks and lack adaptability to these complex environments. We propose Hydra, an adversarial attack framework for AI-driven video streaming and edge security services. Hydra integrates three innovations: Multi-Stage Service Disruption (MSSD) that coherently misleads successive pipeline components, Resource-Aware Transferability Enhancement (RATE) for effective attacks under edge constraints, and Meta-Reinforcement Learning for Adaptive Perturbation (MetaRAP) that dynamically tunes attack parameters based on real-time system states. Evaluations on a multimodal medical RAG system demonstrate Hydra’s superior efficacy, achieving up to 75% ASR against specialized retrievers at 𝜖=8/255, a 12% absolute improvement over baselines. Hydra also exhibits strong transferability, defense robustness, and a favorable efficiency trade-off, requiring only 2.1 seconds per adversarial example. This work highlights a potent threat to edge intelligence services, urging development of robust multi-stage defenses.

When a language-model agent fills tool-call arguments, the argument value is the action: 转一万二 ("transfer twelve thousand") must become transfer(12000), and 两斤苹果 ("two jin of apples") must become order(weight_kg=1.0). We show that today’s LLMs silently corrupt such values when Chinese locale semantics must be converted into canonical arguments — producing schema-valid, confidently wrong actions that no validator catches. We introduce ZhToolLocale, a diagnostic benchmark with machine-computed gold values (lunar-calendar libraries, deterministic numeral and unit parsers), paraphrase triplets, and a pre-registered audit protocol; we report 95% bootstrap confidence intervals and significance tests for every headline claim. Across six models from five families — including two Chinese-trained — we find: (1) lunar-calendar conversion degrades significantly in every model (all p ≤ .031) and catastrophically in five of six (0–28%, vs. ~100% on fixed-date controls); a pre-registered matched-pair control refuted our initial memorization hypothesis — explicit lunar dates are, if anything, easier than festival names — indicating genuine conversion difficulty rather than recall; (2) Chinese-centric training confers a sizable advantage at the large tier (deepseek-chat: 93.3% overall, 81% lunar — the strongest model tested) but does not rescue the 7B tier (qwen2.5: lunar 3%); (3) paraphrase disagreement detects scattered errors (precision up to 0.92) but is structurally blind to consistent mis-conversion, which we quantify; (4) a deterministic converter restores 94.8–100% accuracy with significant gains for three models (all p ≤ .0012), but routing is tier-governed: frontier models route voluntarily (93–99%), a mid-tier model improves only under an explicit mandate, and two 7–8B models cannot be made to route at all (gains non-significant) — implying that deployable-tier safety must be architectural rather than prompted. We release the benchmark as a generator whose golds are computed on demand, with a per-category deployment card.

Large language models have recently demonstrated remarkable abilities to self-correct their responses through iterative refinement, often referred to as self-consistency or self-reflection. However, the dynamics of this self-correction mechanism may differ substantially depending on whether the model is tasked with open-ended text generation or with selecting the most appropriate response from multiple predefined options. In this paper, we conduct a systematic investigation of these two paradigms by comparing performance trends and error-correction behaviors across question answering tasks, covering language models of different scales and families. Our experimental results reveal distinct patterns of improvement and failure modes:

While open-ended generation often benefits from the flexibility of re-interpretation and compositional refinement, multiple-choice selection can leverage clearer solution boundaries but may be limited by the provided options. This contrast also reflects the dual demands faced by emerging agentic LLM applications: effective agents must not only generate and refine open-ended plans or explanations, but also make reliable discrete choices when operating within constrained action spaces. Our findings, therefore, highlight that the design of self-correction mechanisms should take into account the interaction between task structure and output space, with implications for both knowledge-intensive reasoning and decision-oriented applications of LLMs.

Persistent context files (AGENTS.md, CLAUDE.md) are standard practice for guiding AI coding agents, yet evidence for their effectiveness is contradictory. We present a controlled ablation of context-injection strategy across two frontier agents (Claude Code and Codex), 17 real tasks from 3 repositories, and 288 evaluated runs with gold-test evaluation. Context strategy does not measurably move correctness on either agent (bounded to ≤10–15pp via equivalence testing). A failure-mode triage reveals why: agents fail on implementation skill—feature design, pattern selection, exact wiring—not missing repository knowledge that a context file could supply; a manipulation probe confirms the real AGENTS.md never converts a near-miss to a pass on either agent. We further show that borderline task difficulty is agent-specific (Spearman 𝜌=0.75), offering a candidate explanation for prior contradictions: single-agent studies draw tasks from different agents’ informative bands. We release all code, data, and analysis.

LLM agents degrade as trajectories lengthen, motivating current research on context management methods that compress, filter, or restructure history. All of these methods share the untested assumption that context degradation is the primary bottleneck of agentic systems. To test this hypothesis, we introduce an oracle-state intervention framework that decomposes agent failure modes, isolates context as a causal variable by replacing a failing agent’s accumulated history with ground-truth structured state, and measures recovery. Across 420 failed WebArena trajectories with two models (Qwen3.5-27B and GPT-5.2), we find that context management is effective but not in the way commonly assumed in recent literature. In the default WebArena Qwen and GPT configurations, replacing context after failure recovers under 10% of failed trajectories. Additionally, independent human annotators attribute only 11.7% of failures to context-caused errors. The remaining 58–74% of failures persist even after both context and environment are restored, and the convergence between intervention-based and human-attribution estimates suggests that context availability explains only a minority of failures. Yet continuous oracle state throughout the trajectory improves success by 19-38%, and the gap between oracle-maintained and self-generated state widens from 7% to 18% as trajectories lengthen. This causal intervention framework reveals that post hoc context repair rarely rescues failed agents, while continuously maintained privileged state improves long horizon execution. Moreover, our findings suggest that progress on agentic systems will require improving planning and execution capabilities alongside context engineering, rather than treating context quality as the dominant bottleneck.

A coding agent must emit a valid tool call—a parseable invocation of a tool in the provided schema—before the harness can execute its chosen action. We study how local serving stacks affect this protocol step and show that measured outcomes can depend on the serving layer rather than model behavior alone. In Ollama, the default tools= request is gated per model by a static template flag: some models are accepted and return calls as text, some return native tool_calls, while Phi-3 and Gemma-3 are rejected before inference. In our harness, rejection and retry exhaustion are not preserved as structured failure metadata, so downstream analysis can misclassify them as model non-calls and naively report 0% fidelity. Adding a text tool list while retaining the native channel recovers much of the measured fidelity for accepted models, whereas a uniform text protocol reduces fidelity for Llama-3.2, which has native tool-call support. Cross-stack probes on Ollama, llama.cpp, vLLM, and SGLang show different handling of the same request. Constrained decoding removes parse failures but can induce non-termination, and turn-pooled versus per-instance estimates differ by up to about 55 points. We conclude with a checklist for treating serving behavior as part of the evaluation protocol.

Large language models are increasingly being used as agents that interact with tools and environments to carry out complex tasks. This development has prompted a growing range of benchmarks and leaderboards for comparing agent performance. Yet the evidence behind these comparisons is not always fully disclosed. Even when every disclosed record is authentic and correctly scored, a reporter with access to the complete ledger can choose a favorable subset. We study whether rank claims remain informative under such adaptive release. An authenticated disclosure frame fixes systems, denominators, bundle bounds, and the canonical claim universe before selection. Worst-case completion keeps every reported edge sound, but a budgeted reporter can still choose which comparisons become certifiable. This separates the soundness of reported comparisons from neutrality in what gets reported. Across four public agent-evaluation ledgers, target-directed release produces positive attention gaps for most eligible targets at intermediate budgets; the effect depends on budget and bundle layout and disappears under an equal-per-system quota in one ledger. We derive an exact polynomial algorithm for the unit-cost target problem and prove weak NP-completeness under heterogeneous costs. These experiments are structural stress tests, not evidence that benchmark maintainers select evidence strategically. They show that certificate soundness does not imply disclosure neutrality.

Large Language Models (LLMs) typically exhibit a performance profile where reliability degrades as task complexity increases. We address the challenge of generating high-quality natural language executable plans for complex tasks by introducing GRASP, a strategy-aware, multi-stage planning framework. GRASP decouples the planning pipeline across specialized, context-isolated modules: it pre-compiles global macro-guidelines (GenPlan), explores alternative localized strategies within isolated context windows (RevPlan), and independently evaluates trajectories using a multi-criteria discriminator (VerPlan). Empirical evaluations show that GRASP consistently establishes a new state-of-the-art frontier across diverse datasets, yielding substantial accuracy gains over direct LLM planners on Natural Plan Calendar Scheduling (∼12.4%↑), ZebraLogic (∼30.8%↑), and SciBench Math. Crucially, under multi-task scaling—where standard planners suffer immediate performance collapse—GRASP completely flattens the multi-task degradation penalty. In interleaved dual-task environments, GRASP achieves an absolute accuracy gain of up to 16.7% over direct LLM planners. Furthermore, by isolating context and enforcing strict macro-regularization, GRASP outperforms frontier reasoning models (such as GPT-5-mini) by a margin of 14.5%.

A large language model (LLM) can understand a game state in two ways: answer questions about it directly, or write a program that computes the answer. We compare these modalities on three real-time-strategy tasks—factual extraction, strategic assessment, and win prediction—holding states, questions, and models fixed. On mechanical questions, code wins decisively: hit-point (HP) aggregation rises from 30% to 100%, and the direct-mode failures are mostly a refusal to declare ties, not bad arithmetic. On judgment tasks, code seemed to win as well, until a trivial baseline exposed why: on snapshots from bot games the material leader always wins, so sign(ΣHP) alone scores 100%. On contested states built to remove this artifact, no arm—direct, chain-of-thought, or code-based—reliably beats chance (40 tests, Holm-corrected), while a simple unit-count cue reaches 72%. Code makes LLMs reliable aggregators; it does not make them judges. We contribute the shortcut audit and hard-set construction that judgment benchmarks built on game traces should adopt.

In on-device agentic systems, Small Language Models (SLMs) are essential for ensuring privacy-preserving personalization and low-latency execution. However, despite their capacity to handle isolated atomic tasks, during complex multi-step orchestration SLMs frequently suffer from stochastic drift—a phenomenon where minor logical deviations cascade into irrecoverable trajectory failures. In this work, we demonstrate the primacy of pre-execution planning, revealing that mitigating initial logical drift is vastly more effective than mid-trajectory interventions. Building on this insight, we propose FORST(FOundation fiRST) framework that implements sequential logical scaffolding in pre-execution planning, where predefined expert anchors are injected as fixed pre-computed KV-cache states. While the scaffolding ensures logical stability, KV-cache reuse efficiently realizes the structured anchor prefix without repeatedly prefilling static anchors. Then, we can maintain a monolithic reasoning chain within predefined logical planning boundaries and effectively eliminate the redundant computational overhead of structured anchors. Extensive evaluations on the MCPEval benchmark demonstrate that establishing this rigid logical foundation via achieves competitive or numerically higher performance than frontier models, while achieving an latency reduction on edge-device platforms.

Reusing evaluator-derived step rewards across evaluation channels raises a measurement question: does the assigned direction remain stable? We define Agent Step Value (ASV) as a channel-indexed before/after target-margin gain and identify the state-by-channel interaction from a matched 2 × 2 design. Across frozen PubMed question-answering transitions, cross-channel gain signs disagree on 48.3% of comparisons, compared with 0.3% for same-channel retries. Transition interactions are nearly balanced in sign (median +0.014; 10% trimmed mean -0.033), but asymmetric negative magnitudes produce a positive direct mean and a negative generated-view mean under the empirical target mixture. Equal-target weighting preserves a negative mean interaction but not this strict mean reversal. Matched retrieval designs localize the negative component to the generated-view coordinate and retain its direction across three sign checks that vary readout and evaluator stack. In a matched 512-token source-contract follow-up, source-only generation restores the positive mean direction on artifact-bearing retrievals and removes parser-detected support claims from artifact-free before-state views. These results support an explanation in which generated support inflates the pre-retrieval baseline. The resulting audit can localize channel sensitivity and test an intervention before step rewards are reused.

Large language model (LLM) agents increasingly rely on function calling to act across multi-turn interactions, but compact models remain brittle when tool availability, required parameters, and dialogue constraints change over time. Existing synthetic tool-use data often broadens coverage without targeting the recurring failures of a specific model. We present FUSE, a failure-guided refinement approach that uses the model’s own function-calling errors to synthesize harder training scenarios and then emphasizes difficult examples during supervised fine-tuning. This data-centric design turns diagnostic failures into targeted training signal while remaining simple to apply with standard offline fine-tuning. Experiments on BFCLv3 show that FUSE substantially improves a compact Qwen3-based model across standard and stress categories, including missing functions, missing parameters, and long-context interactions. Evaluation on BFCLv4 provides initial evidence that the same refinement strategy can transfer to selected agentic function-calling tasks.

Long-document question answering usually forces a choice between loading the whole document into the context window and bolting on a separate retriever. Agentic AI suggests a broader option, giving the agent the document path and letting it decide how and what to read. Agent Skills, a standard for packaging expertise into folders an agent loads on demand, supply a ready mechanism: progressive disclosure, which exposes only what a query needs, from a short description down to the specific passages. Practitioners rapidly adopted this pattern for book-length understanding tasks, but the evidence to support such choices has been anecdotal. We run the first controlled study of the pattern, comparing raw-document navigation and several designs of Agent Skills packs against a classical hybrid retriever across three agent harnesses and three model families on ∞Bench. On a single book, the gain depends on the harness, running large when the agent navigates the raw document poorly but near zero when a strong agent harness already divides and retrieves on its own. When scaling up to tasks that span many books, raw-document navigation collapses while one-level progressive disclosure degrades more slowly and pulls ahead. A second, deeper routing level never helps and sometimes breaks accuracy outright, so one level is enough. Progressive disclosure buys context, not intelligence: it is redundant while a strong agent can locate the right passages itself, and decisive once the corpus grows too large to navigate by reading.

End-of-trajectory rewards do not identify the decision associated with a tool-agent failure. We treat step-level credit assignment as failure localization and test whether it supports selective repair. Silver labels come only from completed training trajectories. Prefix-only classifiers use task-grouped model selection and remain frozen during evaluation. On the retail domain of 𝜏²-bench, the selected classifier reaches 46.4% out-of-fold F1 and 61.1% F1 on 60 held-out trajectories comprising 333 scored decisions. The frozen gate selects 101 decisions, but the verifier changes actions in only seven trajectories. None changes binary reward relative to its task-matched baseline. Airline and telecom produce no changed trajectory. Mean reward differences are small, and every task-clustered confidence interval includes zero across the three domains. An exact replay audit reproduces 60 telecom baseline trajectories but triggers no intervention. The evidence identifies a localization–repair gap: detecting suspicious decisions does not yield repair without calibrated treatment coverage, grounded alternatives, and controlled attribution.

LLM memory systems often filter facts classified as stale from the current prompt. We study this query-time context filtering, without deleting stored memories, in memory-conditioned question answering. Across three models on a synthetic benchmark, controlled label corruption moves hard-drop from gains of +0.186 to +0.230 at perfect resolution to losses of −0.176 to −0.181 at the lowest tested reliability. Paired targeted interventions show greater degradation from false-drop than false-keep; retaining evidence can still fail when incorrect labels steer generation. On one frozen 102/102 calibration/test query split, empirical policy selection using labeled calibration outcomes reaches 0.655 accuracy versus 0.641 for always hard-drop and 0.662 for test-informed oracle selection. A question-disjoint MemoryAgentBench replication is inconclusive: gain +0.03, 95% CI [−0.01, +0.08]. These results establish a controlled sensitivity mechanism, not generalization to long-horizon agents. Global-accuracy crossovers require gold labels and are model- and task-specific diagnostics. Context filtering should be cost-sensitive and validated on matched labeled data; label-free retention is an accuracy fallback under the tested errors, subject to separate privacy and retention requirements.

As LLMs advance into autonomous agents with tool-use capabilities, they introduce security challenges that extend beyond traditional content-based LLM safety concerns. This paper introduces Sequential Tool Attack Chaining (STAC), a novel multi-turn attack framework that exploits agent tool use. STAC chains together tool calls that each appears harmless in isolation but, when combined, collectively enable harmful operations that only become apparent at the final execution step. We apply our framework to automatically generate and systematically evaluate 483 STAC cases, featuring 1,352 sets of user-agent-environment interactions and spanning diverse domains, tasks, agent types, and 10 failure modes. Our evaluations show that state-of-the-art LLM agents are highly vulnerable to STAC, with attack success rates (ASR) exceeding 90% in most cases. The core design of STAC’s automated framework is a closed-loop pipeline that synthesizes executable multi-step tool chains, validates them through in-environment execution, and reverse-engineers stealthy multi-turn prompts that reliably induce agents to execute the verified malicious sequence. We further perform defense analysis and find that existing prompt-based defenses provide limited protection. To address this gap, we propose a new reasoning-driven defense prompt that achieves the strongest initial-turn protection, cutting ASR by up to 28.8%; however, this advantage erodes sharply under adaptive attacks, and an experience-based defense (ToolShield) proves more durable over sustained multi-turn interactions. These results highlight a crucial gap: defending tool-enabled agents requires reasoning over entire action sequences and their cumulative effects, rather than evaluating isolated prompts or responses. Data and code: https://github.com/amazon-science/MultiTurnAgentAttack.

Long-term memory systems often organize experience before future queries are known, but whether this improves evidence access under a fixed read budget remains unclear. We evaluate inexpensive query-agnostic segmentation and hierarchical retrieval on AMA-Bench agent trajectories and BEAM conversations spanning 128K to 10M tokens. Experiments use exact serialized-context budgets, development-only tuning, two sentence encoders, and uncertainty estimates clustered by trajectory or conversation. In an exploratory post-plan analysis of 1,117 step-anchored trajectory questions, fixed three-turn windows outperform all tested alternative boundary methods. On BEAM, primary-encoder flat retrieval recall falls from 0.564 at 128K to 0.331 at 10M. Single-vector hierarchical routing further reduces recall; multiple local vectors recover part of this loss but do not surpass flat retrieval. Doubling the read budget from 8,000 to 16,000 characters improves evidence recall at every scale with both encoders. However, the 10M evaluation, comprising 130 questions from eight conversations, detects no corresponding QA improvement. Single-vector routing reduces QA accuracy, while multi-vector routing is not distinguishable from flat retrieval. These findings identify coarse representation quality as a bottleneck in the tested hierarchies and show that improved evidence access need not yield better answers.

Personalized agents must decide whether retrieved user memory should be used, ignored, updated, or queried before it affects a task. We develop an empirical audit protocol for structured intermediate outputs: auditing dataset shortcuts, isolating bundled prompt changes, testing answer-associated labels and decomposed evidence, and retaining provider-level execution failures. A 480-example synthetic development set initially suggested large gains from a state-structured prompt bundle, but TF-IDF diagnostics showed lexical separability and no positive standalone Ignore cases. We therefore build a frozen 160-example controlled counterfactual set with 40 matched four-way families and rule-derived reference policies. There, exposing four state definitions improves accuracy, but adding an isolated explicit state-output field does not significantly improve policy accuracy for Llama-3.3-70B and gives only a marginal, non-significant gain for GPT-OSS-120B. Supplying benchmark-associated state labels shifts policy predictions; because those labels deterministically map to policies, this is label-conditioning, not evidence of a faithful mechanism. Family-level and seed-stability analyses show that example accuracy overstates counterfactual consistency. A decomposed semantic-evidence follow-up does not improve routing for the cleanly evaluated endpoint, while the GPT-OSS condition fails provider-side request validation. We evaluate policy classification only.

Selecting the best algorithm for each optimization instance is difficult because algorithm performance can vary substantially across instance structures and parameter settings. Existing learned selectors often encode historical experience in fixed model parameters, making new performance records costly to incorporate. We introduce a memory augmented agent for instance wise algorithm selection based on retrieval augmented generation. Given a new instance, the agent retrieves structurally similar historical cases, each containing the complete performance profile of all candidate options rather than only the winning label. It then reasons over the retrieved objective gaps and rankings before invoking an algorithm and configuration. New verified records can be added directly to the external memory without retraining the selector. We evaluate the proposed approach on 10,898 geographically grounded hub location instances, using 8,718 instances for retrieval and 2,180 for testing across five algorithm and configuration options. The proposed method achieves 68.35% selection accuracy and a 0.549% objective gap relative to the virtual best solver, outperforming the single best solver and retrieval only label voting baselines. These results demonstrate the value of combining retrieval with reasoning over complete solver performance records.

Self-reflective agents typically bundle monitoring language with a downstream control policy, making it difficult to identify why reflection changes behavior. We present a six-condition ablation that separates evidence re-exposure, diagnostic questions, uncertainty labels, differentiated follow-up actions, and diagnosis–action alignment. Our testbed is an LLM forecasting agent evaluated with Llama-3.3-70B on 310 armed-conflict escalation cases, plus supplementary GPT-4o and 12-country transfer probes. Diagnostic questions produce comparable F1 to free reflection (0.296 vs. 0.297), and adding a seven-label taxonomy while holding the action fixed changes F1 by only +0.008. The closest matched comparison for action differentiation is taxonomy-only versus random action assignment (F vs. D), which holds the typed monitor and taxonomy fixed while replacing the shared-action policy with randomly varying substantive actions: F1 rises from 0.304 to 0.338. Type-matched routing reaches 0.379, but we detect no significant advantage over random assignment (C vs. D, p=0.714). This underpowered contrast does not establish equivalence; it prevents assigning the gain specifically to matching. Brier scores remain flat and transfer is unreliable. Thus, richer reflective traces do not establish better agent control. Our results motivate controls matched for information, vocabulary, and actions when attributing gains in reflection pipelines and identify simple action perturbations as a necessary baseline for type-specific policies.

We study whether LLM agents can autonomously synthesize deployment-ready MCP (Model Context Protocol) servers from raw REST API documentation, and whether the resulting servers are reliable enough for downstream agentic task completion. Evaluating 8 synthesis models across 14 APIs under three documentation conditions — full docs, no docs, and counterfactually mutated docs — we find that synthesis is broadly feasible: 11 of 14 APIs achieve sufficient tool coverage, and synthesized servers match or exceed human-authored references. The dominant mediating factor is not model capability but parametric gravity: for well-known APIs (GitHub, Stripe, Shopify), training-time knowledge overrides in-context documentation, rendering documentation provision neutral or harmful. A counterfactual mutation study confirms this: high-familiarity APIs show near-zero adherence to renamed parameters, while low-familiarity APIs show near-complete adherence. Logit-lens probing shows the representations of memory and context become increasingly separable in the final transformer layers, consistent with deep feed-forward MLP modules as the site of parametric association storage. Synthesis coverage (SYNTH) correlates strongly with agent task completion (r = 0.937), providing a reliable pre-flight quality signal without requiring agent evaluation.

Reasoning about safety is essential for embodied agents powered by large language models (LLMs). We introduce the Contextual Action and Plan Safety benchmark (CAPSafe), designed to evaluate safety reasoning at two levels: (1) an action-level dataset that tests whether a single action is safe given its environmental context, and (2) a plan-level dataset that evaluates whether agents can generate safe long-horizon plans for complex hazardous household tasks. CAPSafe models safety conditions where hazards are situational (inferred from object properties), multiple hazards can co-occur within the same environment, and each is annotated with a relative severity level for fine-grained analysis. Evaluating nine closed and open-source LLMs reveals a substantial gap between hazard recognition and safe long-horizon planning. Several models achieve high accuracy in recognizing unsafe actions (exceeding 90% in the best cases), yet safe long-horizon planning remains substantially harder. Gemini-3-pro achieves the highest safe success rate (78.19%), while other models remain below 65%. These results highlight that robust safety reasoning over long action sequences remains an open challenge.

Tool-using agents repeatedly transmit prompts and conversation history, but aggregate token counts obscure which components drive cost. We introduce a five-component accounting convention for append-only agent loops, using provider usage records and fixed-prefix token-count probes without a local tokenizer. With prompt caching disabled, in 954 episodes (4891 requests) from one claude-haiku-4-5-20251001 model across three synthetic task families, the recurring fixed prefix accounts for 76.6% of baseline input tokens. Tool schemas contribute 69.1%, echoed assistant output 6.5%, and the tool-result residual, including framing, 16.9%. A descriptive fit to continuing episodes gives per-step input growth of about 363 tokens; longer-horizon cost ratios are extrapolations. Across the same 144 tasks, family-informed tool pruning reduces mean cost by 24.1% with a success difference of -1.4 percentage points (paired 95% CI [-3.5, 0.0]). A rolling history window saves 2.0% with identical observed outcomes. Truncating results at 600 characters reduces success from 98.6% to 47.2% while more than doubling mean cost alongside longer unsuccessful episodes. These findings apply to the tested model, mock tools, and short horizons. The accompanying package reproduces the analysis from frozen usage logs and prefix counts.

Closed-loop synthetic-data pipelines combine planning, LLM generation, verification, and deduplication; accepted outputs steer later generation. We study a production pipeline for Vietnamese contact-center compliance data and identify feedback blindness: an internal feedback mechanism stops serving its intended function while every stage completes normally, leaving only downstream metric degradation as evidence. We diagnose four instances: cumulative-count eviction empties a recency window; batch-level deduplication misses candidates scored before any batch member is committed; static boundary exemplars repeatedly anchor generation; and a fixed-size anti-repeat reference set loses informativeness as the dataset grows. We describe architecture-level fixes, mechanism-specific health signals, and a controlled ablation and scaling evaluation. The central lesson is that aggregate duplicate and acceptance rates are insufficient for diagnosis; pipelines should also monitor the feedback mechanisms that produce those outcomes.

It remains unclear whether agentic augmentation (iterative tool use, code execution, or retrieved memory) improves over single-pass VLM inference for medical image interpretation. We present a systematic quantitative evaluation of the Recursive Language Model (RLM) framework on a multimodal medical task, comparing three pipeline architectures (P1 direct vision, P2 iterative RLM tool use, P3 model-written code agent) across four retrieval-augmented memory conditions on 100 CheXpert Plus chest X-rays, scoring with CheXbert (14-label classification) and GREEN (radiology-specific LLM judge). Three findings emerge: (1) direct vision outperforms both agentic variants; the RLM loop consumes 7-9× more tokens without adding task-relevant information when its only tool calls the same VLM; (2) retrieval consistently helps, but the optimal pool size depends on metric and pipeline: top-5 is the sweet spot for direct-vision report quality (GREEN +37%), while 24 retrieved cases further benefit pathology classification (CheXbert +50%) and can improve GREEN for agentic pipelines; and (3) qualitative analysis shows code agents with a pretrained classifier succeed when it returns confident pathology scores but fail on normal cases where it returns noise-floor outputs, illustrating that tool quality, not framework complexity, determines outcome.

Enterprise agent language models must coordinate specialist agents over continuous business-event streams, yet most multi-agent evaluations assume discrete request-response workflows. We present a controlled simulation based on a sanitized enterprise research snapshot: 208 scenarios, 393 events, and 1,051 expected agent calls across Persona (<10 agents), Department (20–80), and Enterprise (200) registries. We compare DAG Plan Execute with ReAct and evaluate a minimal Task Manager for priority inference, related-event merging, and preemption. Both systems perform well on the Persona cohort but score lower on the Enterprise cohort, especially for Simple scenarios; a larger, more confusable discovery space may contribute. The tested DAG Plan Execute implementation generally maintains higher agent-call precision, whereas ReAct shows stronger recall and Expected Answer Completeness in several Enterprise conditions. The Task Manager reduces high-priority queue latency by 14–75% and improves related-event Expected Answer Completeness by over 20 percentage points at Enterprise scale. These controlled simulation results characterize the tested systems but do not establish production readiness.

Reliable tool use is a fundamental capability for agentic systems, yet existing evaluations primarily focus on successful tool execution and provide limited insight into how agents recover from tool failures. To address this gap, we introduce ToolMisuseBench, an offline deterministic benchmark for evaluating tool misuse and fault-conditioned recovery. ToolMisuseBench evaluates recovery behavior across domains, fault types, and difficulty levels, using output-grounded success criteria and typed recoverability metadata. The seed-42 release includes 6,000 training, 900 development, and 1,200 public-test tasks covering CRUD, retrieval, file operations, and scheduling, including failures such as timeouts, rate limits, hidden first-call schema perturbations, authorization errors, and adversarial responses. Using the benchmark, we evaluate deterministic reference agents together with local 3-4B tool-calling LLMs (Qwen3-4B, Ministral-3B, and Llama-3.2-3B). On the public-test split, Parser+Repair achieves 0.767 task success and 0.736 conditional recovery success, compared with 0.158 task success when recovery is disabled. On a shared stratified 216-task diagnostic panel, Qwen3-4B, Ministral-3B, and Llama-3.2-3B achieve task success rates of 0.796, 0.903, and 0.773, respectively. Fault-level analysis reveals distinct recovery weaknesses that overall task success alone fails to capture. ToolMisuseBench provides a controlled and reproducible framework for diagnosing tool-use recovery under frozen fault plans, enabling more rigorous evaluation of robust tool-using agents beyond execution accuracy.

After a factual conflict, the same credited answer can arise from different record-choice rules and different active-store states. We audit this ambiguity in MemoryAgentBench’s 400-question FactConsolidation Single-Hop (FC-SH) task. A fixed exact-template parser identifies 282 questions with two target arrivals and distinct values, plus 118 singletons. In every two-arrival case, the larger serial also arrives last, so last-arrival and maximum-serial selection coincide. We cross these selectors with stores that expose either one active record or all versions. Given this alignment and the policy contracts, all four policies produce the same 400 prediction strings and receive 393/400, although the exact earlier fact ID is visible on 0/282 targets under one-record stores and 282/282 under all-version stores. Swapping the two arrivals while preserving their serials separates the selectors on all 282 targets. Under this primary parser, seven gold items are credited only to the earlier value; joining both candidates yields 400/400 under substring exact match. Thus, within this controlled grid, released FC-SH answers identify neither record choice nor active-store visibility. This release-level measurement audit pairs an order contrast for choice with direct store observation for visibility.

Repository-level repair requires not only correct patches but also inspectable records that explain how issue requirements are translated into code changes and post-edit evidence. We contribute THEMIS, a stage-aware repair workflow that externalizes this requirement-to-repair process through semantic interpretation, a runtime requirement–code graph, graph-derived Developer guidance, retained repair rationale and patches, and post-edit audit records. A retrospective audit of 300 SWE-bench Lite cases demonstrates that these artifacts provide broad support for cross-stage inspection: a complete Developer rationale is available for 288 cases, and 214 cases (71.3%) retain a complete audited field set connecting the selected stages. The retained records further enable systematic measurement of cross-stage correspondence: target symbols recur in 62.6% of Developer rationales and in 62.8% of patches, rising to 75.8% when related symbols are included. In a paired 100-case comparison, the relational workflow resolves 19 cases versus 9 for the direct same-input condition, corresponding to a 10-percentage-point improvement. Together, these results show that THEMIS turns otherwise implicit requirement-to-repair transitions into observable traces, enabling systematic inspection of how repair decisions persist, align, and evolve across stages.

AI systems increasingly let one model choose another: an orchestrator routing a task to a sub-agent, or an agent picking a tool from a registry, by self-described ability. What happens when that self-report cannot be checked? In MATCHMARKET, a client hires one of three candidates whose true competence is hidden (2, 5, or 8 of 10), earning the competence of its hire. Candidates are themselves language models, inflating claims under an audit-and-fine incentive. Selectors rarely fall for the loudest liar, picking the inflator only 8–15% of the time. They fail by fleeing to mediocrity: 75–89% of their mistaken hires land on a safe, moderate candidate, not on the liar. A trivial yardstick makes the cost concrete: believing the highest claim earns 6.72 of a possible 8.0, and no selector we test significantly beats it. The best merely matches it, while the weakest fall significantly below (pooled 5.99, paired p = .002; random scores 5.0). A verified record recovers much of the loss, but only for selectors near chance without it, widening rather than closing cross-model gaps. Every outcome is scored arithmetically, no LLM judge. Unverified capability advertisement behaves like a market for lemons whose cost falls on the honest, capable provider.

Multi-agent LLM architectures, such as LangChain and AutoGen, largely assume English as the lingua franca for internal inter-agent communication, even when the end-user task is non-English. We fill this gap by evaluating a two-agent extraction-answer core, with an additional back-translation agent in the English-forced condition, across four typologically diverse languages (Hindi, Chinese, Spanish, Arabic; n=300 per language) using the Aya-23-8B model. We compare a native-language pipeline to an English-forced one (which incorporates a final back-translation step from English to the user’s language). We discover a statistically significant English-Forcing Tax (surviving a strict Bonferroni correction) that isolates the cost of English routing from general multi-agent orchestration overhead. Forcing inter-agent communication through English reduces Exact Match accuracy by 13.0 percentage points (Spanish) up to 30.6 percentage points (Hindi) compared to native-language multi-agent execution. Using chrF scores as a diagnostic measure of English-reference lexical overlap, we find that lower overlap is strongly associated with pipeline failure, consistent with translation loss being an important contributor to the observed performance drop. These findings suggest a compelling case for native-language routing in agent frameworks when the source and target languages are typologically distant, reducing a compounding translation tax.

Multi-agent Systems (MAS) combine multiple model outputs to solve complex reasoning tasks. However, despite rapid growth of available open-source models, there is limited research on how to select optimal model candidates out of this massive pool. We systematically evaluate 8 model selection strategies (including model size, accuracy and answer diversity) across before-generation (routing) and after-generation (majority-voting, LLM-as-a-judge) MAS architectures on challenging scientific benchmarks. Our findings show a significant gap between theoretical oracle potential and actual performance: Expanding candidate pool sizes often degrades performance below that of the top performing base-model. We find that candidate selection within a single model family is the strategy that yields the best relative performance over a standalone model. These results demonstrate that adding arbitrary models to a heterogeneous MAS may introduce system instability, highlighting model selection as a critical design choice for multi-agent systems.

Large language models are increasingly used as interacting agents, but it remains unclear how robust their coordination is when public communication is unreliable. We study this question in iterated N-player Stag Hunt games played by homogeneous LLM groups under controlled programmatic action inversion, which changes both the public transcript and the actions used for execution. Across an experimental grid spanning group sizes, coordination thresholds, corruption levels, and seven LLMs, we observe three main patterns. First, honest agents’ pre-flip Stag choices decline as corruption increases, but the sharp fall in public success is primarily mechanical. In the focal N=5,M=3 setting, pre-flip success remains 78% at 80% corruption, while public success falls to 12%. Second, honest choices are associated with the public history available at decision time, particularly under high corruption. Third, three threshold-style public-report benchmarks yield similar action-match rates to the LLM agents, showing substantial descriptive agreement between LLM decisions and these benchmarks. Overall, our results show that original choices, public actions, and executed outcomes must be separated when evaluating multi-agent robustness, as corrupted communication can severely and predictably degrade mutually beneficial cooperation.

Large language model (LLM) agents have demonstrated the ability to solve multi-step tasks requiring planning, tool use, and external information access, yet growing execution histories increase inference cost and expose reasoning to outdated, irrelevant, or misleading information, potentially degrading reasoning quality. Existing memory approaches organize or compress execution histories but provide limited mechanisms for deciding which memories remain active. We introduce the Weighted Memory Tree (WMT), a hierarchical memory system that organizes execution into tasks, subtasks, and actions while assigning each memory a dynamic retention score. Event-based updates and selection-based decay revise these scores, allowing WMT to preserve useful information, fold completed trajectories, suppress low-utility content, and retain access to folded context. We evaluate WMT on GAIA and its text-only subset, GAIA-Text, using Qwen3-8B, Gemma 4 E4B, and Llama-3.1-8B, with ablations and memory-poisoning experiments. Relative to linear memory, WMT improves accuracy by an average of 9.97 percentage points on GAIA-Text and 10.10 percentage points on GAIA, while reducing prompt-token usage by 32.8% and 32.2%, respectively. In our controlled memory-poisoning experiments, WMT reduces the measured persistence and propagation of intentionally injected unreliable information. Our results suggest that effective long-horizon agent memory depends less on storing more information than on deciding which information should remain active.

Large language models (LLMs) are increasingly being explored in cyber-physical system design as a promising tool to facilitate the conversion of natural-language requirements into formal specifications, and the generation of plans and controllers that provably satisfy them. Existing LLM-based pipelines, however, mostly validate the designed controllers by simulation, and carry no mechanism for improving their outcomes from past designs. We present ACIES, a self-evolving agentic contract-based framework for cyber-physical system control synthesis. Given a set of system requirements, ACIES automatically generates a control architecture that realizes the requirements using a sequence of synthesis episodes. Within each episode, a Translator agent formalizes the system requirements into temporal logic, and a Decomposer agent structures them into a system-level contract and proposes a decomposition into component-level contracts that can be independently implemented. A formal verification step is then integrated in the decomposition loop to check whether the component-level contracts are consistent, compatible, and provide a correct refinement of the system-level contract. Each failed check, at the verification or implementation step, returns a diagnostic signal that drives automated contract repair at the system or the component level. Across episodes, a Summarizer agent distills verification outcomes into reusable lessons stored in a memory. Evaluation results on five control synthesis tasks with eight LLM backbones show that the formalized system requirements are successfully decomposed and mapped to a synthesizable control architecture that realizes them, while self-evolution reduces the Decomposer token consumption by up to 65% and the end-to-end synthesis latency by up to 62%.

Selective routing is useful only when an expensive branch solves cases that a cheaper branch misses. We call this prerequisite expert complementarity and ask whether a capable iterative agent actually provides it. In a prospectively frozen financial question answering study, we compare Static with bounded ReAct across multiple capability tiers. Within each tier, the arms share the model, answer contract, retrieval interface, scoring, and resource limits. A stronger model removes ReAct’s average deficit and brings it to parity with Static, showing that relative performance depends on model choice. Yet Static and ReAct still succeed on nearly the same items. Across 150 shared items per tier, ReAct adds only one or two unique exact successes; oracle quality headroom point estimates are at most 1.7 points, while ReAct takes 2.1 to 2.6 times as long. A stability check preserves this pattern. Capability and complementarity are therefore distinct: improving an expensive branch can make it competitive without making it useful for routing. Evaluation contracts should be harmonized and paired complementarity audited before training a selector. This result concerns bounded ReAct over controlled benchmark evidence, not iterative agents in general.

Research on coding agents mainly focuses on generation quality. Yet, safety-critical automotive software makes two additional demands. It prioritizes verification, since every change is reviewed, traced to a requirement, and checked against industry standards such as MISRA and AUTOSAR before it can be committed. It is also a narrow domain, written against layered architectures, embedded resource constraints, and organization-specific interface conventions that are scarce in the data general-purpose LLMs are trained on. We present a coding assistant for automotive C/C++ that addresses both demands. First, it exposes three tools to the user over the Model Context Protocol (generate, edit, and verify), allowing developers to run verify on any code on demand rather than relying on the agent to check its own output. Second, rather than using a general-purpose model, we customize the underlying LLM for the automotive domain through prompt engineering and fine-tuning. We construct a dataset from automotive production code of AUMOVIO, where each function is paired with the requirement texts and Unified Modeling Language (UML) behavior it must satisfy. In a two-week pilot on active production projects, developers report verify as the most-used tool and confirm using it on their own code and not only on generated output. Comparing model customization strategies, 75% prefer the faster prompt-engineered configuration. MISRA- and AUTOSAR-inspired pattern checkers indicate that prompt engineering achieves a lower number of violations on C, while fine-tuning shows better conformance with architecture patterns for C++.

Running a tool agent n times and keeping the best run is worth 2.19–22.37 points on τ²-bench, a floor any scaffold spending n times the calls has to clear. Choosing among those runs, for no additional model calls, is worth anywhere from −23.68 to +12.28 depending on which rule does the choosing. We evaluate 9 selectors that read only the agent’s own trajectory across 28 model × domain cells and 10 403 released trials, and add an LLM judge that reads the trajectories but never a reward and never the evaluation criteria. Every arm is therefore reward-free: the nine rules are verifier-free as well, the judge is not, and neither sees a label. Against a mean best-of-K oracle band of 12.29 points, which is what a selector holding the reward would collect, the judge recovers 32.7% and the best free rule 21.0%, leaving 67.3% of that band unreachable without the label. One measurable property of the rollouts orders the task distributions, the no-mode rate, the share of tasks on which no two of the K rollouts emit the same write-action sequence, and what it predicts is when agreement is exploitable rather than when selection is possible. Where agreement-based rules are nearly inert the judge is worth +5.95 against +0.53; where the rollouts are undifferentiated it returns 1.19 against a band of 14.4 points; on a third domain it returns −0.64 and loses to a free heuristic costing one fewer call.

Multi-turn LLM evaluation is typically reported as a single win-rate scalar, conflating distinct capabilities. We introduce AIDG (Adversarial Information Deduction Game), formalizing multi-turn adversarial dialogue as a two-player partially observable stochastic game (POSG) and decomposing performance along Seeker (extraction) and Holder (containment) roles. The decomposition isolates three failure modes: cooperative-prior leakage, constraint–reasoning interference, and inefficient hypothesis-space traversal. Across 439 games over six frontier LLMs, defensive performance is tightly clustered (σ = 1.9 ELO) while offensive performance varies substantially (σ = 53.3 ELO); confirmation framing increases extraction odds 7.75× over uninformed deduction (p < 0.00001); and constraint violations account for 41.3% of deductive failures, uncorrelated with scale (ρ = 0.0). We position the containment-over-extraction gap not as a surprising finding but as a measurable consequence of locally-resolvable defensive decisions versus globally-coupled offensive planning, and use the decomposition to attribute the gap per model. All design choices, including turn-decay weighting and the Bradley–Terry rating model, are derived from explicit assumptions.

Large Vision-Language Models (LVLMs) are increasingly explored for multimodal medical diagnosis, their ability to conduct accurate consultation remains challenging because of unstable visual reasoning and limited tracking of patient information across dialogue turns. To address this problem we present V-RAM (Visual Retrieval-Augmented Multi-Agent Reasoning Framework), a training-free framework for improving LVLM-based multimodal medical dialogue diagnosis. V-RAM improves medical dialogue diagnosis by adding visual retrieval, structured memory, and multi-agent reasoning around a pretrained LVLM. It retrieves similar medical images as diagnostic cues and tracks patient evidence across dialogue turns. The framework operates at inference time, it requires no supervised fine-tuning and can be used with different LVLM backbones. Experiments on benchmark show that V-RAM outperforms existing baselines and the current LVLMs by 7.3% on an average with 86% lesser number of turns that are used for medical dialogue diagnosis.

Long-term conversational AI requires two distinct capabilities: remembering what was discussed and adapting to how a user’s preferences evolve over time. We argue these operate on different timescales and require separate but interoperable architectures. AgenticLOCI is a five-agent framework built on this premise: session retrieval fuses a short-term semantic ranker with a long-term keyword scorer via rank fusion, while personalization is maintained through a versioned deliberation-state memory that accumulates decision-episode snapshots across interactions. We evaluate on three benchmarks using Qwen 2.5-7B-Instruct. On LoCoMo, the AgenticLOCI Backbone achieves F1 = 0.342 versus BM25 = 0.285 while consuming 7.3× fewer input tokens per item. On LongMemEval-S, item-level analysis confirms that the two retrieval layers solve architecturally distinct sub-problems; at 7B scale the keyword layer leads, a reversal expected to resolve at larger model sizes where LLM ranking is more reliable. On PersonaMem, all configurations perform within a narrow band, establishing that memory benefit is conditional on context-budget pressure. The longitudinal personalization layer is evaluated on a synthetic mobility benchmark with known user trajectories, keeping 97.6% of stable preference fields unchanged and making warranted updates with 58.3% precision against an ablation baseline. We demonstrate the full framework as a personalized urban mobility assistant for Magdeburg.

Conversational agents are good at recall. But recall only reacts and an agent that answers only what it is asked can never raise what the user never thought to mention. We study the other direction which we call prospective memory. A background agent keeps a small set of pins in the assistant’s context before any question arrives and each pin must be retired by the right authority. A constraint from chat, such as a stolen bike, ends only on the user’s word. One from a city feed, such as a suspended tram line, ends only on the feed’s end message, whatever its forecast said. We release a benchmark of 60 days, ten users and 1,802 graded decisions in a mobility assistant, with automatic answer keys for pinning, forgetting, recommendation and habit discovery. Capture, at least of the templated reveals this benchmark renders, is the easy half. A dense 8B model captures events almost as well as a 397B/17B sparse one and what separates the two is forgetting. The small model hoards and invents. The large model’s one systematic failure is a release cascade, asked to rewrite its whole pin table, it treats any release signal as a reason to shorten it. Asked instead to name its removals as explicit operations, it names only the right ones, every swept pin survives, and required session recall rises from 0.77 to 0.99, from 0.80 to 1.00 on the events both runs captured.

Hiring is bilateral: employers assess fit, while candidates present and defend evidence of their qualifications. Yet résumé screening—the first gate—is commonly automated as a static, one-call judgment over a résumé–job pair. We study a two-agent alternative in which employer-side and candidate-side agents represent these roles, exchange evidence, and update their judgments before deciding who advances. We compare procedures on 600 constructed résumé–job pairs using GPT-5.5 and Claude Opus 4.7. Two-agent screening advances more applications (33.3% to 39.3% for GPT-5.5; 34.0% to 35.5% for Opus 4.7). Across three runs on the common 191-pair borderline pool, pass-instance rates rise from 4.5% to 26.2% and from 6.5% to 16.1%, respectively. This is not a uniform relaxation: two-agent screening rejects applications one-call advances, changing decisions in both directions. At similar pass volumes, the procedures advance different applications, and no one-call threshold recovers applications consistently selected by two-agent screening. Among discovery-selected cases re-executed in fresh runs, two-agent-only selections recur less often than shared selections—clearly under GPT-5.5, less certainly under Opus 4.7—while a separate one-call follow-up shows no comparable decline. As hiring becomes agent-mediated on both sides, the screening procedure—not only the model behind it—shapes who reaches human review and how reliably that access recurs.

People increasingly hand fact-finding to LLM agents that search the web and return a cited report. We ask whether the belief a user states when making the request changes the research itself. We give 48 expert-verified health claims to four search-equipped agents under believer, skeptic, and neutral framings, analyzing 1,874 of 2,064 registered runs plus second-round arms reported separately. The effect is asymmetric: stating belief leaves conclusions unchanged (0.02, claim-clustered 95% CI -0.04 to 0.09), while stating doubt moves them toward rejection (-0.26, CI -0.39 to -0.15). Across opposing framings the same agent’s conclusion shifts 0.26 points on a five-point scale (CI 0.15 to 0.40), a quarter of one label step, and every system moves the same way. The shift sits on contested claims (0.61) and is small on the settled tiers (0.10 and 0.11). Strictly opposite conclusions were rare and their registered paired test was null (p=0.250), so this is a shift in degree rather than a reversal of verdicts. The trajectories mirror it: skeptic framing more than doubles deny-seeking queries (net -9.9 points against +0.5 for believer framing). Crossing the retrieved evidence between framings splits the gap into how the agent reads what comes back (0.23, CI 0.12 to 0.34) and which evidence comes back at all (0.08, CI 0.00 to 0.16), two channels that add to the end-to-end gap by construction and put about three quarters of it on reading, though the evidence interval reaches zero. Labeling retrieved results against the claim finds no evidence that stated doubt returns a more hostile pool (1.7 points, CI -1.5 to 5.5). An indicative post-hoc follow-up on the 32 claims it reached adds a both-sides instruction to the skeptic condition, where the bias lives, and shifts conclusions by 0.16 (CI 0.04 to 0.30). We release FramedFacts-48: claims, prompts, and analyzed runs.

Agentic retrieval-augmented generation (RAG) interleaves retrieval, reasoning, and answer generation across multiple hops. A retrieval error at hop 1 can surface only as a wrong answer at hop 3, while later retrieval can also repair the trajectory. This paper introduces AgenticRAG-FP, an interventional benchmark for causal failure attribution in agentic RAG. The benchmark injects a certified fault at a specified hop, re-executes the downstream trajectory, and evaluates diagnosers against the known intervention site. Its central question is whether a post-hoc trace still identifies the injected hop after the suffix changes. In the dense Claude Haiku 4.5 structural sweep on three-hop MuSiQue questions, coverage-based diagnosis is 0.91 at hop 1 and 0.00 at hops 2 and 3 (n=43, 36, 21 failed trajectories), but it predicts hop 1 in 96% of failures, an earliest-hop rule; at hops 2 and 3, an LLM judge and frozen-hop repair localize only 0.25 to 0.48 of failures. In larger content-corruption runs, depth-2 exact-hop accuracy is 0.00 for coverage, 0.49 for an LLM judge, 0.59 for frozen-hop repair, and 0.14 for suffix regeneration (n=51); the judge and frozen-hop repair do not significantly differ (exact McNemar p=0.33).

Agent quality on long-horizon tasks is usually reported as a single number: the aggregate gain of a tool-using agent over a non-agentic baseline on a benchmark. We argue that this number is not a property of the agent. It is jointly determined by a property of the benchmark and by which sub-component of the agent happens to be load-bearing, and reporting it alone makes agent quality look benchmark-dependent and irreproducible. We propose a four-stage evaluation protocol that separates the two. Stage 1 characterizes the benchmark before any outcome is observed, using an LLM-free diagnostic we call recoverability: whether evidence missed on the first retrieval is still reachable from what the system has already seen, and if so whether it is reachable by reformulation or only by structural navigation. Stage 2 assigns credit with an ablation ladder — one-shot RAG, a choice-free fixed-iterative loop, and a full agent — that localizes any measured gain to iteration or to open-ended agency. Stages 3 and 4 add the validity checks and cost accounting that change verdicts in practice. Applying the protocol to about 6,000 logged agent episodes over eight corpora and three reader families, we find that the headline agent gain ranges from −0.21 to +0.30 (each corpus in its native metric) purely as a function of the recoverability regime, and that on reformulation-recoverable corpora the choice-free loop recovers 65–96% of the gain normally credited to agency under the two stronger readers (and most of it under the open reader). Agency earns its cost only on the longest-horizon, navigation-recoverable corpus, where iteration saturates by round 2 and leaves a residual (+0.15 under the open reader at full N) that no round budget closes under any reader.

The reliability of Large Language Models (LLMs) for API calling degrades in multilingual settings. A common failure occurs when a model selects the correct tool but generates argument values in an inconsistent language, which we term Argument Language Mismatch (ALM). Although semantically correct, such outputs are operationally invalid and not captured by standard API-calling metrics. We revisit post-training strategies for mitigating ALM and find that, in our benchmark, supervised fine-tuning (SFT) provides a strong baseline, substantially improving argument language consistency and end-to-end function call accuracy. Under consistent model selection, SFT achieves performance comparable to, and sometimes exceeding, more complex reinforcement learning (RL) approaches. We further examine whether RL with structured, argument-aware rewards offers additional benefits. While methods such as Group Relative Policy Optimization (GRPO) can improve language consistency and better preserve general reasoning ability, these gains are incremental and most pronounced in generalization and multi-objective trade-offs. Overall, our results suggest that much of the performance in multilingual API grounding can be achieved through careful supervised training, with RL providing fundamental improvements.

We audit whether outperforming random activation directions establishes improved agent per- formance. Using Llama-3.1-8B in a molecular identification task, we compute every legal action’s exact probability of eventual success un- der optimal continuation. An entropy-weighted steering direction outperforms all 39 layer- and norm-matched random controls on 42 held-out molecular pools in a frozen comparison. Posthoc audits reveal the limits of this result. An untuned policy that chooses uniformly among legal actions also outperforms every random control, with no detectable difference from the steered policy; uncertainty intervals still admit modest effects in either direction. The steering direction ranks 10th among 36 directions when controls are matched to its construction, and its original-bank rank falls to 3/40 when pools sharing chemical scaffolds are grouped and weighted equally. Executed episodes show no detectable improvement over the unsteered baseline, with both policies performing near random under constrained decoding. These findings concern one model and task; they do not establish that steering cannot improve a competent agent. They demonstrate that su- periority to a random-direction bank does not by itself establish policy improvement. We propose five reporting controls covering trivial baselines, direction construction, inferential units, state selection and horizon, and executed behavior.

Simulated students play a central role in training and evaluating AI tutor agents, yet most approaches rely on static personas or unconstrained generation and implicitly assume stationary behavior. Real student responses also depend on transient learner states such as affect, attention, and memory that evolve during interaction. We introduce DYSAS, a dynamic state-aware student-simulation framework with three tutoring-facing modules, Affect (A), Attention (T), and Memory (M). Their low, medium, and high levels form the discrete ATM27 state space. We operationalize this interface through Dynamic Learner States (DLS), a prefix-only representation inferred at each turn without future leakage and used to condition student generation. We evaluate static and dynamic variants on the Eedi benchmark across prompting, supervised fine-tuning, and preference optimization. No configuration wins every objective as each of correctness-state match, misconception behavior, social tone, lexical fidelity, and tutor-response compatibility favors different settings. We therefore present DYSAS as an audit mechanism within a multi-objective simulator-selection framework, not as a universally superior add-on.

Large Language Models (LLMs) are increasingly serving as autonomous agents, and their utilization of external tools via the Model Context Protocol (MCP) is considered a future trend. Current MCP evaluation sets suffer from issues such as reliance on external MCP services and a lack of difficulty awareness. To address these limitations, we propose MCPAgentBench, a benchmark based on real-world MCP definitions designed to evaluate the tool-use capabilities of agents. We construct a dataset containing authentic tasks and simulated MCP tools. The evaluation employs a dynamic sandbox environment that presents agents with candidate tool lists containing distractors, thereby testing their tool selection and discrimination abilities. Furthermore, we introduce comprehensive metrics to measure both task completion rates and execution efficiency. Experiments conducted on state-of-the-art LLMs reveal significant performance differences in handling complex, multi-step tool invocations. All code is open-source at https://github.com/Brunestuder/MCPAgentBench.

We audit how action interfaces and recovery mechanisms shape the evaluation of local open-weight Reflexion agents on 134 ALFWorld valid-unseen tasks. The analysis covers six primary admissible-action conditions and historical free-text runs across three model deployments. We organize the audit around four implementation contracts: action execution, recovery triggering, treatment changes, and outcome construction. Historical free-text runs show substantial invalid-output fallback, while designated admissible-parser fallback remains below 1.1 percent; this contrast does not by itself establish parser faithfulness. Structured Recovery Prompting (SRP) combines a recovery prompt with counter resets and relaxed termination limits. Its implementation-reported task-success differences range from 0.00 to +6.72 percentage points and are smaller than the baseline contrast between the two Qwen model sizes. The available evidence supports descriptive comparisons rather than an isolated recovery-prompt effect. The audit identifies incomplete repeat detection and termination-based success labeling, highlighting the importance of trigger coverage and outcome definitions when interpreting recovery results. The contribution is a focused diagnostic audit that connects aggregate performance to implementation behavior and provides concrete checks for evaluating recovery in the studied agents.

Agentic pipelines for structured-query generation are rapidly expanding, but it is unclear which part of the loop produces the gain. We use LAST-CQ—a five-agent, training-free, execution-grounded Text-to-Cypher framework—as an instrumented testbed, running three counterfactuals over 2,471 live-database queries and six backbones spanning three vendor scale tiers. Removing correction is worth between 3.1% aggregate execution-BLEU against the single-pass system and 12.3% against a no-refinement counterfactual (up to 80.7% for the weakest backbone). Replacing schema-grounded, LLM-synthesised feedback with raw database error strings costs almost nothing (20.9% vs. 19.9% naive exact match; <0.2% end-to-end; equivalent within ±0.075 set-F1 by two one-sided tests). Spending the same call budget on parallel sampling degrades quality by 10–11%. What works is detecting failure and routing it to a retry, not the feedback sophistication or number of samples. LAST-CQ itself recovers 91.7% of queries that fail under single-pass generation, while a query that succeeds first time still costs exactly one LLM call. We also show that n-gram overlap on serialised results is not a bound in either direction: it over-scores against set equivalence on 65.9% of results while under-scoring against judged semantics. Finally, we calibrate our LLM judge against blind human labels and find it optimistic by 9 points.

Tool schemas can validate each language-model-generated call while still allowing quantity identity to disappear during a longer workflow. A model may read 250 milliseconds, checkpoint a bare scalar, and later pass 250 to a destination expecting seconds. We call this loss of executable unit and identity metadata semantic scalarization. We present DIMLOCK, a small runtime that carries quantities as opaque references, resolves exact compatible conversions, and rejects cross-dimensional references before destination execution. We also introduce a frozen stress protocol with logical checkpoint save/load, same- and cross- dimension distractors, and receipt-backed scoring. Across 36 workflows and 540 controlled fault trials, DIMLOCK contains all 108 primary faults while prose and structured scalar handoffs contain none. A same- dimension identity swap defeats every interface, confirming the boundary of dimension-only enforcement. Native MCP pilots cover 216 Codex trajectories and a balanced 36-trajectory Claude Sonnet replication; tested models usually rejected explicit traps, so we make no prevalence claim. The result is a focused runtime invariant and a reproducible test for a failure class that local tool schemas do not express.

LLM agents in social simulation revise their opinions implicitly, in context: how open an agent is to persuasion can neither be specified nor verified, and collective outcomes inherit the model’s training prior. We introduce Bayesian Chronicle Agents (BCA), a minimal belief layer separating what an agent believes from how it speaks. Each stance is a probability, updated by one Bayesian step per utterance heard. A single prior-strength parameter 𝜅 encodes stubbornness, modeled after its role in Friedkin–Johnsen (FJ) opinion dynamics. We then sweep this parameter to yield three canonical regimes of opinion dynamics on demand (consensus, persistent disagreement, committed-minority influence), with persistent disagreement matching the FJ closed-form fixed points at R² = 0.93–0.99. We further show that prescribed 𝜅 remains recoverable after the language round-trip, with perfect rank-order recovery across all four models. Explicit belief also makes simulation auditable: the layer surfaces systematic per-model stance biases that end-to-end simulation would silently absorb.

Agent evaluation today depends on per-trace LLM-judge inference or human review, too expensive to run on every trace; production systems fall back to sampling a fraction. We find that failing agents leave a detectable behavioral signature in standard observability telemetry: disproportionate effort relative to outcome. We formalize four behavioral failure signatures from agent telemetry, validated on 4,671 traces spanning five agent domains: Compensatory Effort (17–159% more effort on each domain’s primary signal, all five domains, p<0.006, the dominant signature every- where tested); Output Inflation (+38% larger patches on SWE-rebench; direction and reliability vary by domain); Unfocused Orchestra- tion (+5.98–12.3% higher token entropy, four of five domains); and Correlation Decoupling (effort-signal correlations shift structurally between pass and fail traces, p<0.001, three domains). A simple threshold check over these signals, with no judge, no label, and no model call, already catches 52.9–61.5% of failures. Across three human-labeled benchmarks, judges miss 6.5–18.1% of confirmed failures on the two directly comparable ones, and a trained telemetry classifier (AUROC 0.821) recovers 61.5–75.4% of each judge’s misses; combining judge and classifier cuts the miss rate from 18.1% to 4.4%, a 4.1× reduction against the weakest of the eight judges tested (GPT-4o-mini); stronger judges start with fewer misses, so the reduction is correspondingly smaller.

In large-scale enterprise settings, centralized multi-agent systems (MAS) are increasingly adopted, in which a coordinator delegates user requests to lightweight, domain-specialized sub-agents. While this architecture improves modularity, scalability, and cost efficiency, its reliability depends not only on accurate routing but also on sub-agents’ ability to calibrate their responses to capability constraints. In particular, sub-agents built on smaller fine-tuned models often struggle with such calibration, leading them to over-answer ambiguous, underspecified, misrouted, or unsupported requests and produce hallucinated outputs instead of actionable feedback. To address this challenge, we present EARS (Explanatory Abstention for Reliable Sub-Agent Modeling), a production-oriented framework that reframes sub-agent abstention as an inter-agent communication protocol: a sub-agent does not merely abstain, but exposes an actionable failure state to the coordinator. EARS curates human-agent interaction data using an ensemble of calibrated LLM-as-a-Judge models, producing structured abstention labels and rationales under a taxonomy of sub-agent failure modes. These data are used to fine-tune sub-agents to detect failure conditions and return rationales for coordinator-level clarification, rerouting, or fallback. We evaluate EARS in a large-scale production e-commerce assistant supporting enterprise business intelligence workflows. EARS improves the overall response pass rate from 68.5% to 78.9%, demonstrating that sub-agent-side explanatory abstention improves MAS reliability.

Natural language-to-graph query generation (NL2GQL) requires schema-aware reasoning and multi-hop traversal planning, posing graph-specific challenges beyond the extensively studied Text-to-SQL setting. Existing NL2GQL systems typically solve each query in isolation, limiting their ability to learn from prior executions. We present ExGQL, an experience-guided framework for self-improving NL2GQL. ExGQL combines hierarchical memory, experience evolution, and capability synthesis to distill execution feedback into reusable reasoning strategies and convert validated strategies into executable capabilities. We further introduce a Failure-Directed Curriculum (FDC) for a separate practice phase, in which recurring failure patterns guide the selection of tasks that address the system’s current weaknesses. We evaluate ExGQL on NL2GQL-Bench, a benchmark of 400 expert-annotated graph query tasks over a production-scale graph database. ExGQL achieves an average accuracy of 0.869 versus 0.662 for the strongest evaluated baseline (+31% relative improvement), while reducing average query-generation time by 28%. In a separate practice phase, FDC reaches comparable accuracy of 0.861 with 60% fewer practice episodes and 36% lower runtime than fixed-order practice. These results show that NL2GQL systems can improve by reusing execution-derived experience, while failure-directed practice makes experience acquisition more efficient.

LLM agents can make unsafe tool calls even when instructed to behave safely. Existing defenses constrain agents before execution, modify tool inputs/outputs, or rely on LLM judges; these approaches may depend on model behavior or block unsafe actions without helping the agent recover. We argue that the execution environment should instead enforce safety as the agent runs and steer it toward safe alternatives when violations occur—we call this Environment Steering. We implement this by modeling the agent and harness execution state as database tables, track the record-level data flows, and check these data flows against declarative policies during runtime. When violations are detected, policy- and context-specific feedback steers the agent toward safe trajectories. On AgentDyn, this enables the agent to improve task success rate over no-defense while achieving 0% attack success rate.

Majority-vote LLM ensembles improve accuracy monotonically with ensemble size when agents are homogeneous samples of the same prompt (Li et al., 2024). We test whether this holds for four heterogeneous clinical-role prompts on one frozen MedGemma backbone, across binary tumor detection, tumor subtyping, MS, and stroke triage. Our comparison is against single-agent triage: we ask whether role-diverse voting clears that bar, not whether it matches sampling diversity, since we do not run a paired homogeneous-sampling forest. On a paired 500-image evaluation per task, a four-agent forest does not reliably beat single-agent triage: significantly worse on binary tumor (both stages) and MS end-to-end, indistinguishable on tumor subtyping and stroke. Pairwise error correlation between roles is far from independent on every task (φ = 0.43–0.75 vs. φ = 0 under independence), explaining why voting fails to average errors out. Roles diverge behaviorally on the easiest task but collapse onto a shared “normal” bias on the two hardest, so voting cannot recover sensitivity no member has. Dissent rate, a candidate escalation signal, is valid on two tasks, uninformative on MS, and inverted on stroke, where agreement is less reliable than disagreement. These results argue against assuming role diversity on a shared backbone helps by default, and for task-conditional validation before deployment.

Whether a language model looks demographically biased can depend on how the audit asks its question. A charitable-aid benchmark reports that the same models favor minority applicants when rating requests one at a time and penalize some when ranking side by side. We test whether that reversal generalizes to hiring, lending, and medical triage: 40,726 requests to five models, applications differing only in the applicant’s name, and a primary test fixed before collection. It does not. None of 36 planned contrasts survives correction. The rating advantage keeps its sign at roughly half the published size, and a precision extension bounds any hiring ranking penalty below the published effect, though the lending and triage ranking floors sit above that margin, so the exclusion is conclusive for hiring ranking and for rating in all three domains only. Planted disparities tracking their injected sizes and a directional replication on the original aid materials bound these nulls. The audit is livelier than the demographics: models recognize transparent audits nearly always, tie every identical-content comparison whether the varying detail is race or a hobby, and reward first-listed candidates as much as any demographic effect we measure. Audit verdicts reflect audit construction more than demographic bias.

Large language model (LLM)-driven evolutionary search is a powerful paradigm for automated program and algorithm discovery, yet existing systems are largely memoryless: each run explores from scratch, so agents repeatedly rediscover the same improvements and re-encounter the same dead ends. We introduce SEDIMA, a persistent hierarchical insight memory for evolutionary search agents. SEDIMA distills raw traces into natural-language insights, clusters them by semantic similarity using attention-weighted centroids, and retrieves relevant guidance to condition future mutations, accumulating transferable knowledge across runs and problems rather than within a single trajectory. As a drop-in module that leaves the search operators unmodified, SEDIMA improves average final performance by 5.5% on AlgoTune and 6.6% on ALE-Bench LITE under a fixed budget of 100 evaluated candidates. Under OpenEvolve, SEDIMA requires 32.3% fewer iterations on average to reach baseline-best performance across the five evaluated backbones.

LLM-based critics are widely deployed as safety guardrails for agentic systems, yet their reliability under realistic operating conditions remains underexplored. We demonstrate that critics calibrated against a fixed system snapshot degrade as tools are added and application domains expand, with no external signal indicating weakened detection capability. We formalize this problem along controlled axes and introduce DRIFT (Detecting Reliability Issues From Tool-scaling), a benchmark for measuring critic robustness under environment change. Building on SafeToolBench’s multi-domain tool ecosystem, we construct verified cross-app composition scenarios, both violating and compliant, where policy violations emerge only from data flow across application boundaries, and evaluate 9 models across 6 provider families on a two-dimensional scaling matrix separating intra-app noise from cross-app noise. We find that (1) all 9 models degrade under cross-app scaling, with UPR increases ranging from +0.083 to +0.296, (2) tool scaling shifts critics toward reduced discriminability, missed violations rise while false blocks simultaneously fall, and (3) cross-app noise consistently dominates intra-app noise, with cross-app deltas positive for all nine models (sign test p = 0.004) while intra-app deltas are directionless (4 of 9 positive, p = 1.0). DRIFT provides a controlled methodology for measuring critic robustness as tool ecosystems evolve, and our results argue for monitoring UPR and FBR jointly rather than aggregate accuracy alone.

Large language models can now operate a web browser and carry out multi-step shopping tasks. Does such an agent leave a behavioral signature in a session that separates it from a person, even when it drives a real browser and presents an ordinary client fingerprint? We study the question at the session level. We first show that weak per-action evidence aggregates into a session-level distinction whose error decays exponentially in the number of actions. We then define six families of behavioral signals, each grounded in a documented difference between goal-directed automation and human browsing, and combine them in a classifier we call HABITS (Human–Agent Behavioral Invariants from Traffic at Session-level). Labeled agent sessions are scarce, so we describe a simulation methodology for driving language models through shopping tasks and instantiate it on public data with realism-controlled planner agents. We evaluate on five public benchmarks spanning e-commerce clickstreams, web-task demonstrations, and mouse dynamics, reimplementing prior methods on the identical corpus. Against agents that imitate human timing, timing-only classifiers fall to 0.35 recall at a 1% false-positive rate and to 0.01 -0.02 when timing is fully matched, while HABITS reaches 0.96 on the clickstream corpus. Paired tests separate HABITS from every restricted-view baseline, and equivalence tests show that strong classifiers given the same features are interchangeable, which places the contribution in the behavioral view rather than the estimator. A check against real agent trajectories supports the structural assumption our simulator relies on and finds one realism parameter set too favorably.

Existing coding-agent benchmarks test functional correctness—whether code runs and returns expected outputs. Silent scientific methodology bugs pose a different challenge: they execute without error yet produce plausible, scientifically wrong results. We introduce BioSciReview-Bench, a benchmark of 200 manually reviewed synthetic bioinformatics code snippets (100 matched correct–buggy pairs) spanning five domains and six bug categories. Each snippet is presented in isolation without construction metadata; agents must detect whether a bug exists, localize it to a specific displayed line, and classify it by type. We evaluate six configurations across three model families and two agent harnesses. All achieve detection F1 of 0.94–0.99, but performance drops with stricter requirements: joint localization within three lines reaches 0.92–0.98 F1, and full diagnosis requiring exact line plus correct category falls to 0.74–0.89 F1, a gap of approximately 10–23 percentage points below detection. Filtering and quality-control bugs have the lowest average Full Diagnosis@±3 success rate, and agents sometimes confuse them with normalization errors. Detection performance is high on this controlled benchmark, while delivering actionable diagnoses remains substantially harder.

Tool schemas constrain the structure of a call but need not determine how request evidence becomes argument values. We formalize this layer as a field-level binding contract over presence, evidence, ordered operations, parameters, and validation, and state an identifiability boundary: no schema-only estimator can guarantee recovery of policy distinctions absent from its input. We audit all 1,311 filtered requests in pinned Berkeley Function-Calling Leaderboard (BFCL) V4 Live Simple and Multiple with five open checkpoints and the official abstract-syntax-tree (AST) checker. A partial structural diagnostic and AST acceptance are non-nested: 1,470 of 6,555 calls pass only the diagnostic and 50 pass only AST. A stratified AI-only audit finds coarse concern detection far more consistent across two isolated passes than fine attribution. A pre-specified test of a deterministic schema-derived operation-label encoding gives 67.70% AST acceptance versus 68.27% under the same guidance without field tags (-0.56 points; 95% task-cluster CI -1.17 to +0.05; 186 fixes, 223 breaks). Value-only turnover is nearly balanced; the net imbalance sits in key-set changes. The point estimate reverses sign on one post-hoc out-of-panel checkpoint; thus no improvement is established on the fixed panel, and independently authored contracts remain untested.

High aggregate scores can make language agents appear socially responsive even when they merely repeat a prompt-conditioned action. We introduce an identifiability-first framework that separates observed outcomes from evidence of adaptation. Across 3,465 released iterated-prisoner’s-dilemma simulations, 94.5% never change joint regime, leaving reciprocity largely unidentifiable. Controlled histories across nine models reveal opponent-dependent responses that vary substantially across equivalent interfaces. The released checkpoints have the smallest opponent-recency effects. Only the intervention cohort shows reliable propagation after a scripted defection. The diagnostic transfers to longer histories, a multiplayer game, and released-interface wrappers, although model rankings do not. Outcome-only evaluation conflates regime selection with adaptive behavior. Separating opportunity, response, and robustness prevents stable outcomes from being mistaken for social dynamics.

Agentic autonomy is essential for large language models (LLMs) to function as agentic AI systems, yet existing benchmarks often emphasize operational aspects, or HOW an agent behaves, such as tool use. The ability to reconsider the user’s underlying purpose, or WHY the agent acts, remains underexplored. We define this capability as "purpose-reflective autonomy" and introduce CaseInterviewBench, an interactive, interview-style benchmark inspired by consultant recruitment. The benchmark contains 104 predefined scenarios, each with public background context, an initial request, hidden facts, and ground-truth problem formulations and resolutions. The target LLM asks questions to uncover hidden facts, then identifies the actual problem and proposes a resolution, enabling us to observe how it infers, revises, and refines its understanding of the scenario’s underlying purpose in a structured setting. We evaluate hidden-fact disclosure, final problem formulation, and resolution using rule-based metrics and LLM-as-a-judge assessments. Across several frontier and open-weight models, we found substantial room for improvement. We also observed severe violations in which some models fabricated facilitator responses during the benchmark flow, potentially creating serious risks in agentic execution.

LLM-based coding agents are increasingly used in software development, but their use in industrial robot programming remains constrained by vendor-specific APIs, organization-specific code assets, and on-premise deployment requirements. We study a bounded ReAct agent that uses local retrieval over API documentation and code examples to generate collaborative-robot programs from natural-language requirements.

This work is a case study of a structured programming domain in which domain knowledge must remain within the organizational boundary and local models may be sufficient. We evaluate seven open-weight models from 7B to 32B parameters, with five independent runs per model, and include one commercial model as a reference. We also apply the same framework to a smaller public Universal Robots set.

Under our agent and evaluation setup, the strongest local 27B model achieves a mean task success rate of 89.6%. Smaller models often use more steps and tokens because they repeat retrieval or fail to convert retrieved evidence into code. These results suggest that model selection should consider task success, interaction efficiency, and run-to-run consistency.

LLM agents that repair failures often discard successful corrections, forcing later episodes to rediscover similar solutions. We study whether finalized repair outcomes can improve subsequent Text-to-SQL episodes without parameter updates. We introduce MERIT, a training-free agent that maintains an online dual-polarity memory of oracle-verified corrections and observed unsuccessful directions. Under oracle-assisted benchmark feedback, only memories from earlier finalized episodes are eligible for retrieval. A deterministic classifier assigns a coarse failure type, which conditions a hybrid lexical-dense retriever before the frozen model generates each revision. Using Qwen2.5-7B-Instruct with identical initial predictions and repair budgets, MERIT improves execution accuracy over stateless iterative repair from 66.34% to 69.79% on Spider and from 47.35% to 48.44% on BIRD. Paired analyses provide clear evidence for the Spider gain but weaker evidence on BIRD. MERIT is not reliably separated from untyped dynamic retrieval on either benchmark, while Reflexion-style memory reaches 51.24% on BIRD at substantially higher inference cost. Ablations show that negative memory contributes modestly, the value of type conditioning and lexical-dense ranking is dataset dependent, and schema-local experience provides the most consistent benefit. These results clarify when causal cross-query memory improves repair and when broader memory representations remain preferable. Our implementation is available here: https://github.com/nhatkhangcs/merit

Large language model responses are non-deterministic, so failures in LLM agents are hard to reproduce: a failure depends on inference that is not bitwise reproducible, on tools that read changing state, and on a multi-step trajectory that a re-run rarely repeats. Record-and-replay makes a run reproducible, but existing agent tooling records runs only to trace or score them, not to test a code change against them. We present Chronicle, which records an agent run at its non-deterministic boundaries as immutable envelopes and replays it from the record. Its central operation, cut-point replay, serves a chosen subset of boundaries from the record and executes the complementary subset live with new code, turning a recorded incident into a regression test that runs in continuous integration. On a benchmark of 6 recorded failures with simulated model boundaries, recording adds 23 μs per crossing (0.008% of an assumed 300 ms model call), full replay issues zero model calls and is bit-stable across 20 repetitions, and cut-point tests fail on faulty code and pass on guarded and benign changes for all 6 incidents. In a mutation study of the guarded tools, cut-point tests catch every mutant that lets the recorded unsafe action through, while a baseline that stubs every boundary, using the same assertion, catches none. Chronicle and the benchmark are publicly available at https://github.com/theagentplane/chronicle.

Long-horizon coding agents can make many edits before testing a task-wide assumption about an API, authentication mode, environment, or persistent state. A late verifier then exposes that the implementation rests on a false premise. We call the missing step contract acquisition and propose Contract Before Code (CBC), harness-level middleware that compiles historical failures into executable preflight gates. CBC normalizes trajectories, extracts recurring failure signatures, and compiles contracts containing an applicability condition, a probe template, and a planning consequence. At inference time, it retrieves applicable contracts and runs instantiated probes before broad implementation work. Among 1,421 parsed failed SWE-Marathon trajectories, weak labels identify candidate rates of 2.6% overall and 17.5% on stripe-clone; these are not root-cause prevalence estimates. A 120-trajectory human audit reports 76.3% contract precision and 71.4% executable-probe synthesis. Across four service/API-clone tasks and 32 runs per method, CBC achieves 31.3% success (95% CI 18.0–48.6), an observed 21.9-percentage-point increase over vanilla, with 55.6% fewer pre-verification edits at 1.11× normalized token cost. The small end-to-end study supports preliminary within-family benefits; broader transfer and the stability of these gains remain unestablished.

Deep-research agents conduct long-horizon investigations through iterative search, evidence evaluation, belief revision, and synthesis. However, they may commit to claims before sufficient evidence is available, causing later reasoning to reinforce an incorrect interpretation. We introduce DeepRewind, an additive control layer for reversible deep research that represents the agent’s evolving epistemic state as a typed graph of sources, evidence, claims, hypotheses, assumptions, commitments, plans, and drafts. Before accepting an intermediate conclusion, a prompt-based world model predicts its impact and estimates reversibility based on hypothesis narrowing, information loss, recovery cost, and contradiction-trigger coverage. A binary controller blocks risky commitments, while a consistency monitor performs dependency-aware rollback when later evidence invalidates them. Across DRBench and LiveDRBench, DeepRewind improves insight recall by 3.6 percentage points and reduces premature commitments by 59.1% relative to Open Deep Research.

Multimodal embodied systems require translating human intentions into interpretable and coordinated behaviors across heterogeneous modalities. However, existing multimodal agents often rely on implicit representations, limiting controllability and cross-modal consistency. We present a hybrid multi-agent framework for interactive multimodal behavior simulation that bridges semantic reasoning and physical execution through a Unified Parameter Representation (UPR). LLM-based reasoning agents transform multimodal inputs into UPR, which encodes behavioral states and interpretable control parameters for simulation agents generating synchronized 3D motion, spatialized soundscapes, and environmental behaviors. We develop BirdsongChat as a prototype implementation of the proposed framework, using interactive avian behavior simulation as a testbed that tightly couples motion, vocalization, and environmental context. BirdsongChat is evaluated on text- and image-guided scenarios involving species, behaviors, affective states, environments, and multi-bird interactions. The system achieves normalized scores of 94.4% for cross-modal coherence, 100% for affective consistency, and 92.6% for generation consistency. These results demonstrate that an explicit intermediate representation effectively bridges semantic reasoning and physical execution, improving controllability and multimodal synchronization. The proposed framework thus offers a generalizable design principle for embodied AI systems requiring interpretable semantic-to-physical coordination across modalities, with potential applications in bio-inspired ecoacoustics, swarm robotics, virtual environments, and creative multimedia.

AI agents are increasingly being adopted in enterprise and personal settings with access to emails, databases, documents, and other tools where they can read, update, and disseminate sensitive information. Much of prior research on data leakage risks in agents has focused on adversarial data exfiltration through prompt injections and jailbreaks. However, sensitive information may also be exposed during non-adversarial use, thereby creating leakage risks even when users issue benign requests. We report a joint evaluation by the Singapore AI Safety Institute and the Korea AI Safety Institute examining agent data leakage in 12 realistic, non-adversarial tasks, spanning areas like customer support, DevOps, web automation, and enterprise and personal productivity, covering five risk types: lack of data awareness, audience awareness, policy compliance, data minimization, and access-boundary awareness. Both institutes tested a common set of scenarios mirroring real-world deployments. To do so, they implemented independent testing pipelines with ReAct-style agent scaffolds, model-simulated users, MCP-based tool environments, and task-specific LLM-judge rubrics. Across the three tested agents, none achieved fully correct and fully safe execution across all scenarios. Successful task completion often coincided with data-handling failures like accessing unnecessary information or disclosing information to inappropriate recipients, indicating that capability and data-handling safety should be evaluated separately. Qualitative review also revealed claim-action mismatches, simulation-aware behavior, user-simulator role reversal, and interpretation gaps in automated judging. Overall, the results indicate that operational data leakage is a first-order agent-safety concern distinct from adversarial exfiltration. This work also provides a methodology for future evaluations and highlights good practices in agentic testing including creating realistic test environments, LLM-simulated human users, and granular evaluation criteria to improve LLM-judge performance.

Optimizing product listings for generative search requires language agents not only to rewrite effectively but also to continually self-improve by retaining successful strategies across diverse categories. However, existing approaches often treat structured listings as free-form text, leaving effective strategies implicit in prompts, search trajectories, or rewritten outputs rather than accumulating them as a reusable agentic memory. To address this limitation, we introduce GLOBE, an agentic framework that drives continual self-improvement by storing optimization knowledge in an evolving handbook of explicit, field-scoped behaviors. In each episode, an Editor applies relevant behaviors and receives rank-based feedback, which a Reflector converts into behavior updates. Crucially for dependable execution in real-world settings, a Moderator removes redundant proposals, a Curator commits accepted changes, and consistently harmful behaviors are deactivated. This cycle transforms episodic feedback into a persistent, field-scoped optimization memory. Experiments show that structured behaviors outperform monolithic prompt optimization, while feedback-driven evolution and moderation further improve effectiveness and reduce downside risk. The learned behaviors also remain effective on unseen product categories, suggesting that agentic memory evolution supports interpretable and transferable product optimization.

As large language models are increasingly deployed as autonomous agents, optimizing their behavior in agentic reasoning and tool-use environments remains challenging due to the stochasticity of environment trajectories and associated high reward latencies. Existing automated prompt optimization (APO) paradigms struggle in these regimes: discrete evolutionary methods (e.g., GEPA, ACE) lack parametric memory, while instance-level reinforcement learning (RL) approaches suffer from severe gradient variance and reward collapse on small, human-curated datasets. To address these limitations, we introduce COMPASS (Critique-guided Optimization of MetaPolicies for Agentic SystemS), a sample-efficient dual-agent framework that shifts optimization toward task-level metapolicies and demonstrates that an RL-based prompter policy can be effectively trained on such tasks. To overcome sparse feedback, COMPASS introduces a Critique Experience Buffer that couples scalar rewards with dense textual critiques. This mechanism allows an active “Prompter” policy to internalize corrective linguistic patterns and amortize experiential self-reflection directly into its weights. Empirical evaluations demonstrate that COMPASS stabilizes training and yields up to 2.4× faster convergence. On complex reasoning (BBEH) and tool-use (𝜏-bench) benchmarks, our framework improves success rates from 55% to 90% and 74% to 91%, respectively, while naturally discovering specialized algorithmic strategies.

Procedural memory for language agents typically accumulates as trajectories, scripts, or task-specific skills, so memory grows with the number of interactions rather than the number of distinct procedures learned. We propose a workflow-centric procedural memory framework that continually consolidates repeated experience into a bounded set of natural-language workflows. These workflows are merged online under outcome-conditioned update policies. On ALFWorld and ScienceWorld, our memory is competitive with trajectory-level procedural memory on task success while using 3.5× fewer memory tokens on average (1.1-9.1× across settings), showing that representation besides construction, retrieval and updating is also a key design axis for scalable procedural memory.

Stateful agents increasingly leverage execution logs, tool outputs, feedback, and summaries from prior interactions to inform future behavior. Despite this, safety evaluations typically evaluate agents using either a singular fixed prior history or an empty baseline, via a uniform history rendering. We argue that such evaluations only illustrate a conditional snapshot of model behavior and fail to demonstrate whether these conclusions generalize across plausible prior histories and history renderings. We use the Inspect Evals agentic misalignment benchmark, finding that among eight benign prior histories, a single Opus 4.1 history significantly and repeatedly increased harmful-action rates, whereas the remaining seven alternatives exhibited no such increase relative to the empty history condition. While additional rollout replications improved statistical confidence, they did not improve the representativeness of the overall sample. Conversely, all eight Qwen3-30B histories resulted in an average decrease of 26.3 percentage points in observed harmful actions. Furthermore, we examined four distinct history renderings of 15 benign prior histories across 900 Opus blackmail rollouts. History renderings with tool results and neutral summaries reduced mean harmful-action rates by 10.0 and 15.0 percentage points, respectively. These results emphasize the necessity for evaluations to incorporate diverse, plausible prior histories and history renderings to establish that their findings generalize.

Modern LLM coding agents—Claude Code, OpenHands—share a common inefficiency: they spend most of their token budget finding the file to patch, not patching it. On SWE-Bench Verified, a 30B OpenHands agent averages 23 rounds and 631K tokens per resolved issue, dominated by grep, glob, and view_file calls. No open, systematically-studied agent-style code retriever exists at scale to shorten this phase. We introduce CodeGrep, a 14B retrieval agent trained end-to-end with GRPO to issue multi-turn parallel grep/glob/read tool calls and return candidate files to a frozen downstream coding agent.

On all 500 SWE-Bench Verified instances, CodeGrep preserves the downstream resolve rate (25.8% → 27.0%, +1.2pp) while delivering a much larger efficiency gain (−15% rounds, −19% tokens on resolved instances). Across three retrievers, downstream utility follows a precision threshold: BM25 (precision 0.375) degrades the agent, Jina (0.445) is neutral, and only CodeGrep (0.677) crosses the threshold at which retrieval begins to buy efficiency. To enable this study, we mine supervision from 67K open-source agent trajectories (CATM) and build a Git-worktree environment for multi-turn agent RL. In our setting, applying the efficiency signal at the advantage layer, rather than the reward layer, holds KL drift to one-third and translates cleanly into downstream efficiency. We will release the model, training pipeline, RL environment, and evaluation harnesses.

Large language model agents are often guided by reusable natural-language skills that provide task-specific instructions. Existing work automatically refines these skills by using LLMs to propose and repeatedly incorporate textual edits from agent feedback. We find that the resulting performance depends substantially on which proposed edits are combined, and adding an edit can even degrade the skill. We therefore introduce SkillCombiner, which generates candidate edits and explicitly searches over their combinations. Across various settings, SkillCombiner matches or exceeds a strong baseline in mean performance while reducing optimizer cost by 75.2% and total optimization-stage cost by 21.5% on average. These results show that edit combination is an important optimization target in agent skill refinement.

Agent models are frequently evaluated one decision at a time, where the model predicts the next action based on the gold interaction history, which is scored against a reference. We investigate whether improvement under this protocol is predictive of improved autonomous workflow execution. We study pre-SFT and supervised fine-tuned (SFT) Qwen3 models at 4B and 14B parameters and Gemma 3 models at 4B and 12B parameters on multi-turn customer-support workflows. We find that SFT consistently improves text-turn success, and that overall next-turn success increases for every model under gold-history evaluation. However, these improvements do not transfer to autonomous workflow execution. Tool-specific gains also vary across metrics and models. None of the four SFT models succeeds under holistic workflow evaluation, with strict trajectory completion reaching at most 10.4% workflow success. Our results show that next-turn evaluation is not a reliable proxy for workflow success, motivating separate reporting of text quality, local action correctness, tool execution, and end-to-end task completion.

Autonomous GUI agents have achieved near-perfect task completion on touchscreen benchmarks by tapping directly at target coordinates. However, smart televisions, game consoles, and set-top boxes require users to navigate with a directional pad (D-PAD), turning element selection into a planning problem over an unknown focus-transition graph. We study autonomous task completion on streaming TV devices, where an agent navigates 68 streaming applications using only D-PAD inputs to perform four tasks: login, reaching subscription plans, content search, and settings navigation. We formalize this setting as a partially observable Markov decision process and compare two architectures using three open-weight vision-language models (Google Gemma 3 at 4B and 27B parameters, and Alibaba Qwen3-VL at 235B): (i) a VLM that maps screenshots directly to D-PAD actions, following the paradigm that dominates current benchmarks, and (ii) a system that maintains an explicit model of focus transition dynamics via Bayesian belief updates and graph-based path planning. The dynamics-aware system improves login success for every model tested, by a mean of 8.8 percentage points, and for the Gemma models its relative advantage widens on tasks requiring extended navigation. The benefit is inverse to model scale, ranging from 16.2 points for the 4B model to 2.9 points for the 235B model. A 4B model with dynamics awareness (32.4%) outperforms a 27B model without it (26.5%), indicating that transition knowledge can outweigh a 6.75×increase in model parameters.

Reasoning substantially improves language models’ ability to solve difficult problems, but long reasoning traces make these gains expensive. Efficient reasoning should express useful analysis in fewer tokens by reducing repetition and text that adds little value to solving the problem. Progress in proprietary models motivates this goal, yet their hidden scratchpads limit direct study of how it is achieved. We investigate whether open-weight models can generate less verbose reasoning by changing the form of their own earlier thoughts, without retraining. Our intervention rewrites prior reasoning into compact state records before pending tool results are revealed, then compares continuations from original and compact histories. Across the tested models, accepted compact histories usually shorten future reasoning; a coding case study also shows less repeated verification after a shared candidate solution. In software-agent deployments, compaction can reduce total spending, although task quality and rewrite-inclusive time depend on the target and compressor. These findings identify reasoning history as one way to reduce future reasoning-token use and motivate model-specific compaction policies evaluated for both efficiency and retained capability.

Multi-agent large language model (LLM) systems increasingly combine model-generated judgments for forecasting and collective decision-making. However, homogeneous agent systems, i.e., those that share a model and evidence, may produce correlated forecasts, and communication may further contract the diversity that remains. We examine these issues in live LLM prediction polls. Across 189 resolved snapshots, mean residual-error correlation was 0.910. Semantic prompt differences were associated with forecast differences, but diversity-aware selection did not detectably outperform prespecified random ensembles. Standard deliberation removed approximately twice as much disagreement benefit as it added through improvements in individual forecasts at the point estimates, with no detectable improvement in aggregate accuracy. In a prospective intervention, a probability-blind summary of forecast-relevant claims retained 93% of initial dispersion, compared with 29% under standard exchange.

Multi-strategy question-answering agents often produce several plausible responses to one query, but must ultimately show the user only one. Voting treats answer agreement as trust, yet strategies can disagree for different reasons, and identical answers can rest on evidence of unequal quality. We cast this final choice as diagnosis-first evidence-risk arbitration and introduce the Evidence-Risk-Aware Strategy Reranker (ERSR), which canonicalizes a pool of six responses, classifies their agreement type, and applies a type-specific evidence rule while returning one unchanged response. Calibration uses only development data; for each evaluation item, trusted reference answers, trusted reference evidence, and utility stay hidden, and the selector never abstains. On a 900-item development split drawn from FEVER, HotpotQA, and SciFact, ERSR improves evidence-aware utility over a development-selected fixed strategy by +0.052, with the largest gains on label conflicts and unanimous answers with uneven evidence. These results suggest that response-set structure, not agreement alone, is a useful signal for choosing trustworthy answers.

Evaluating research in flexible electronics requires weighing evidence across multiple technical and manufacturing dimensions. However, current LLM-based approaches largely rely on unstructured text and generic evaluation criteria, limiting their ability to assess scientific work at the device level. We present FlexSciEval, a multi-agent framework that first turns each paper or research plan into an auditable structured record, then uses dimension-specific agents to score academic impact, fabrication realizability, and novelty. We also construct a 60-paper expert-annotated benchmark for validation and scale the framework to 4,158 papers. FlexSciEval reduces overall mean absolute error by 10.4% compared with direct LLM evaluation on the benchmark annotated by experts, although gains vary by model and dimension. These results indicate that grounding LLM evaluation in structured and dimension-specific evidence improves the reliability of automated scientific assessment in the field of flexible electronics, providing a practical basis for more reliable agent-based evaluation in experimental device research in the future.

LLM-based social simulations are primarily evaluated for behavioral fit, testing whether agents reproduce the actions or response distributions of the people they are simulating. However, the promise of simulation extends beyond behavioral fit. Simulations can explain human behavior, diagnose barriers, and compare large-scale interventions. These use cases depend on understanding why people acted a certain way, not just what they did. As a result, behavioral fit is insufficient for these types of claims because behavior underdetermines the reasoning process behind it. For instance, the behavior of staying silent may be due to disinterest or suppressed speech, and not answering a call may be due to distrust of the caller or limited phone access. In this paper, we propose representational adequacy as a new evaluation target for LLM-based social simulations. By leveraging LLM reasoning traces, representational adequacy measures whether a simulation’s scenario–reasoning–action triples preserve the reasoning process behind the behavior in a way that is faithful to the population and scenarios being simulated. We distinguish representational adequacy from interpretability and alignment metrics, propose ways to integrate it into simulation research, and pose its measurement as an open problem.

When LLM agents exchange messages in a pipeline, abstractive compression discards task constraints. Routing tags get absorbed into prose, privacy fields get paraphrased, hard instructions get softened. Even exhaustive reminder-based prompting retains only 53% of required routing markers, as measured by RTD: constraint tokens are low-salience relative to task content and are dropped at generation step boundaries even when restated in full. We propose C2-FAIC (Contract-Claim Failure-Aware Inter-agent Compliance), which guards the handoff moment directly: it maintains task requirements in a contract ledger outside agent context and checks each candidate message before it enters the shared trace. On 916 task-metric slices across AgentCollabBench, C2-FAIC is strongest on tracer retention and cross-context leakage control, while also reducing instruction decay and claim propagation relative to no intervention. Critic agents remain stronger on pure IDR and CPR process scores, showing that contract enforcement and generic critique address different failure structures. Multi-agent reliability requires controlling what crosses communication boundaries, not only evaluating what the pipeline ultimately produces.

Limited controlled evidence exists on how training data, adaptation method, and model scale jointly affect tool-calling performance in language-model agents. We evaluate supervised fine-tuning (SFT) with LoRA, reinforcement learning (RL) via Group Relative Policy Optimization (GRPO), and SFT followed by GRPO across six Qwen3 models from 0.6B to 32B parameters, covering both in-distribution performance and cross-dataset transfer. SFT with LoRA is the strongest in-distribution method throughout the 0.6B–32B range and best in 15 out of 18 experimental settings. On cross-dataset transfer, the methods are closer: GRPO wins 29 out of 54 settings where training and test datasets differ, but its margin over SFT averages under one point, and SFT→GRPO is rarely strongest at either comparison. Dataset mixing gives consistently strong transfer while staying close to specialized in-distribution training, regardless of method. Additional analysis further confirms that LoRA outperforms full-parameter fine-tuning, demonstrating that LoRA better preserves pretrained agentic behavior.

Driven by the development of persistent, self-adapting autonomous agents, equipping these systems with high-fidelity memory access for long-horizon reasoning has emerged as a critical requirement. However, prevalent retrieval-based memory frameworks often follow an incremental processing paradigm that continuously extracts and updates conversational memories into vector databases, relying on semantic retrieval when queried. While this approach is fast, it inherently relies on lossy abstraction, frequently missing contextually critical information and struggling to resolve queries that rely on fine-grained contextual understanding. To address this, we introduce D-Mem, a dual-process memory system. It retains lightweight vector retrieval for routine queries while establishing an exhaustive Full Deliberation module as a high-fidelity fallback. To achieve cognitive economy without sacrificing accuracy, D-Mem employs a Multi-dimensional Quality Gating policy to dynamically bridge these two processes. Experiments on the LoCoMo and RealTalk benchmarks using GPT-4o-mini and Qwen3-235B-Instruct demonstrate the efficacy of our approach. Notably, our Multi-dimensional Quality Gating policy achieves an F1 score of 53.5 on LoCoMo with GPT-4o-mini. This outperforms our static retrieval baseline, Mem0 (51.2), and recovers 96.7% of the Full Deliberation’s performance (55.3), while incurring significantly lower computational costs. [Code will be released upon acceptance of the paper.]

Evaluation remains a critical bottleneck for interactive agent development. Existing evaluation methods often rely on static benchmarks, which fail to capture the dynamic, multi-step nature of agentic behavior and struggle to expose meaningful failure modes. While user-simulation-based evaluation offers a promising alternative, existing simulation frameworks suffer from two major limitations. First, they provide limited mechanisms for evaluating the quality and comprehensiveness of simulated interactions, making it difficult to assess whether a simulator sufficiently explores an agent’s capabilities and failure modes. Second, most frameworks are restricted to either UI-only actions or API-only actions, limiting their ability to model the full range of realistic user behaviors. To address these limitations, we propose VISTA, a Versatile Interactive user Simulation Toolkit for Agent evaluation.

Recent studies increasingly employ large language models (LLMs) for neural architecture search (NAS); however, they typically rely on in-context learning, struggle to turn past evaluations into accumulated knowledge, and often suffer from structural hallucinations. Motivated by the observation that a search process should systematically accumulate past experience to guide future exploration, we propose CoDA-NAS, a cooperative dual-agent framework that investigates natural language as a medium for structural knowledge. At the core of our framework is a pattern knowledge base (PKB) that extracts, validates, and maintains design patterns from evaluated architectures. These patterns are retrieved as soft priors to guide the searcher agent instead of memoryless generation. The searcher agent employs a structured reasoning constraint to tightly align the generated topological outputs with these semantic concepts, thereby mitigating hallucinations. To ensure empirical reliability, an integrated lightweight predictor agent provides independent numerical validation. By applying a t-test, statistically verified knowledge enters the PKB, effectively curbing the spread of incorrect rules. Experimental results show that under strict search limits, CoDA-NAS achieves competitive performance using only a local open-weights model. Furthermore, the extracted text-based patterns generalize well to new tasks, establishing natural language as a reliable and robust medium for transferring architectural knowledge.

Multi-agent LLM systems combine multiple inference calls, but prior work often confounds how calls are connected with how they are diversified. We study these factors independently: inference topology and source of inter-agent diversity. In a controlled 2 x 3 matrix, we cross parallel aggregation and sequential refinement with stochastic sampling, role prompting, and learned QLoRA specialization, under a fixed three-call budget and output protocol within each backbone. Using Qwen2.5-14B-Instruct and Llama-3.1-8B-Instruct, we evaluate all six configurations on multilingual low-resource emotion detection across nine languages. Parallel learned specialization is strongest on Qwen at 52.83 Macro-F1 and reaches 52.94 on Llama. On Qwen it also exceeds same-backbone zero-shot, few-shot, CoT, and seven-call self-consistency baselines. The preferred topology depends on diversity source: sequential refinement helps stochastic and prompted settings, while the learned Width advantage shrinks from 2.83 points on Qwen to 0.17 on Llama. Depth-wise analysis suggests that later learned specialists can overwrite correct early predictions, although the aggregate effect is backbone-dependent. Overall, how agents are differentiated produces larger performance shifts than topology, which should be evaluated jointly with specialization.

Real code-editing requests typically mix the target program with other related materials, such as test results or documentation. However, models often make mistakes by incorrectly changing these supporting materials or missing the actual code that needs correction. To investigate this issue, we created multiple task scenarios where the code needing changes and the intended behavior remained constant, while only the surrounding supporting texts varied. This approach allowed us to test if supporting texts that resemble code can confuse the model, making it difficult to identify the correct code to modify. To address this, we developed a simple yet effective training method by showing the model examples of correct and incorrect edits. Specifically, Correct examples only modify the intended code, whereas incorrect examples mistakenly alter or duplicate supporting texts. Our results suggest that reliable code editing depends on the model’s early ability to distinguish between guidance materials and the actual code requiring changes.

Despite recent advances, LLM-based web agents still struggle with limited exploration, omission of critical steps, and sensitivity to task constraints. Prior work suggests that many of these failures stem from weaknesses in planning, yet the impact of alternative natural language plan representation remains unexplored. To address this, we introduce , a static planner-executor framework that evaluates the impact of plan representation in agent performance. We first automatically categorize WebArena tasks into 3 difficulty levels, enabling consistent difficulty grading without human annotation. Then we systematically evaluate 4 different plan representations on the tasks categorized as hard: sequential subgoals, narrative, pseudocode, and checklist; across different families of multimodal LLM powered agents (OpenAI, Alibaba, and Google). To account for stochastic variability, we introduce a novel decomposition of agent Success Rate (SR) into 2 interpretable evaluation metrics: Achievement Rate (AR) and Solved-Task Consistency (STC). Our results show that both, the plan formulation and the underlying LLM generating the plan, significantly influence web-agent robustness and task success. Code and dataset are available [here](https://github.com/alzambranolu13/plan-ahead).

LLMs have improved at executing well-specified chains of reasoning but remain weak at conceptual ideation, which identifies what is worth reasoning about in the first place. We cast this as bilevel optimization over (i) a compact hypothesis space 𝛩⊆𝛩₀ and (ii) a point 𝜃∈𝛩, and derive a domain-agnostic prompting strategy — Generate-𝛩 — that makes hypothesis-space construction an explicit pre-solving step. On two challenging puzzle domains, Generate-𝛩 matches or exceeds the max-thinking variant of the same base model (Claude Opus 4.6 or DeepSeek-v3.2/v4) and outperforms reasonable prompting baselines. These results suggest our method works well in tasks that are well-represented in the training set but nonetheless require creativity to solve new instances. Based on our formalism, we propose extensions for solving less familiar tasks as well. Results may provide insight into strategies learned by “reasoning” models during training. Our prompting strategy (or extensions thereof) could be integrated with RL training to produce better exploratory data for reasoning models in ideation-heavy tasks.

Agent benchmarks evaluate policy compliance but assume each policy determines a unique correct action. Natural-language policies can violate this assumption through silence, ambiguity, or contradiction, admitting multiple defensible readings that a single gold trajectory cannot capture. Auditing two 𝜏²-bench domains, we develop a taxonomy of such policy loopholes and show that affected tasks produce unreliable scores: they lower scores across different models in different ways and make every model less consistent across repeated trials. A cross-domain comparison reveals that exploitability requires both policy ambiguity and tool permissiveness: when policy complexity exceeds what tools can enforce, agents resolve gaps inconsistently and scores become unreliable. Policy specification quality sets the ceiling on evaluation quality. Benchmark developers should audit policies before collecting gold annotations.

Tool retrieval is a necessary first step for tool-augmented LLMs once the set of available tools grows to thousands or more, and itscentral difficulty stems from two gaps. A user query states what the user wants to accomplish in task-centric language, whereas tooldocumentation describes functionality, parameters, and calling conditions in function-centric terms. This low lexical overlap produces aQuery-Documentation Mismatch. Complex queries also bundle several functional needs at once, while a single tool usually covers only one,producing a Multi-Intent Query Mismatch. We present ToolFuse, a training-free framework that addresses each gap on a different side ofretrieval. On the tool side, Complementarity-based Semantic Fusion treats tool documentation and LLM-generated synthetic intent as adocumentation view and an intent view of the same tool, encodes them separately, and assigns each tool a fusion weight from the geometriccomplementarity between its two views in vector space, without fine-tuning the frozen dense encoder. On the query side, QueryDecomposition splits a multi-intent query into focused sub-queries, retrieves for each, and aggregates the results with rank-aware round-robin. On ToolBench, the full ToolFuse raises the Per-Split average NDCG@5 of Qwen3-Embedding-0.6B from 64.72 to 75.49, with consistentgains across three additional dense encoder backbones.

We present Trace2Ground, a large-scale web GUI grounding dataset and benchmark built by parsing HTML/DOM to obtain candidate interface elements and using large language models to simulate interaction traces and natural-language instructions. Existing datasets primarily target instructions solvable by matching the target element’s visible text, while real-world GUI tasks may require grounding under user scenarios, duplicate candidates, and relational descriptions. Trace2Ground targets these under-explored stress regimes through three curated splits: PragmaMind, where instructions describe user scenarios without naming UI elements; ContextTwin, where multiple elements share identical visible text and the correct target must be selected using interaction context; and RelateLoc, where the target is specified via spatial or structural relations to nearby anchors. We construct the dataset via an automated pipeline that extracts candidates from HTML/DOM structure, synthesizes traces and instructions with LLMs, and applies multi-view visual checks. Strong specialist GUI grounders underperform on these regimes despite reaching around 90% on ScreenSpot, exposing a gap not addressed by existing GUI training data. Standard supervised fine-tuning on Trace2Ground improves the base model’s performance on our curated regimes by 42.3% on average and yields zero-shot transfer to standard GUI benchmarks, including a 9.8% average improvement on the non-web ScreenSpot-Pro benchmark across six desktop domains.

Recent advances in Vision-Language Models (VLMs) have motivated the development of multi-modal search agents that can actively invoke external search tools and integrate retrieved evidence through multi-step reasoning. While promising, existing approaches typically rely on large-scale supervised trajectories or expensive reinforcement learning (RL), leading to high training cost, instability, and a severe cold-start problem for standard VLMs. We propose a training-free paradigm to empower VLMs with autonomous search capabilities via cross-modal model merging. By fusing a text-based search agent with a base VLM, we show that multi-modal search capabilities can be effectively composed without any additional multi-modal training data. To mitigate parameter interference during cross-modal integration, we introduce Optimal Brain Merging (OBM), a saliency-aware merging algorithm that identifies task-critical parameters based on their impact on model loss using only a small set of calibration samples. Extensive experiments on search-intensive benchmarks (e.g., InfoSeek, MMSearch) reveal that: (1) Model merging secures a reasonable performance floor as a zero-shot agent, with OBM achieving superior search rates; (2) OBM significantly raises the performance ceiling as a warm-start strategy, achieving faster convergence and higher peak accuracy than standard VLM initialization.

Tool retrieval over large API catalogs is a core bottleneck for LLM agents: user queries arrive in colloquial, often underspecified language, while the catalog uses technical API vocabulary that no fixed encoder can bridge on its own. The two dominant training approaches, contrastive encoder fine-tuning and HyDE-style query expansion with a frozen LLM, address this problem from opposite ends and fail in complementary directions: the fine-tuned encoder excels when the query’s surface form already matches the catalog but collapses when it does not, while zero-shot HyDE is more robust to underspecified queries yet generates catalog-unaware hypothetical descriptions that degrade retrieval when queries are well-formed. We introduce CoHyDE, an iterative procedure that trains the dense encoder and the LLM rewriter as a single co-evolving system: the encoder is retrained with InfoNCE on catalog-style hypothetical descriptions produced by the rewriter, and the rewriter is preference-aligned via DPO against the encoder’s retrieval scores, with both sides warm-started on the tool catalog before the loop begins. On a 10k tool subset of the ToolBench catalog, three rounds of CoHyDE improve over the strongest single-component baseline by +2.5 pp NDCG@5 on standard queries and +6.3 pp on held-out vague queries, with gains as large as +8 pp on the hardest vague tier. Ablations confirm that co-training is the key ingredient: using either component in isolation fails to match CoHyDE on both well-formed and vague queries, with losses of up to -8 pp on vague queries.

To achieve complex real-world goals, AI agents must be able to plan appropriate sequences of actions from a task specification. However, evaluating and improving the planning capabilities is challenging because planning involves multiple steps that are difficult to disentangle when viewed as a whole. A more feasible approach is to focus on Temporal Execution Order (TEO), a core planning sub-task that requires partially ordering actions based on dependency by reasoning about temporal and causal dependency constraints. Existing research studying models’ reasoning over temporal execution is limited to automatically derived annotations, approximations of TEO as a linear chain, or text-only inputs. To address this gap, we introduce MATEO (MultimodAl Temporal Execution Order), a multimodal benchmark designed to assess and improve the temporal reasoning abilities of Large Vision Language Models (LVLMs). We acquire a high-quality multimodal recipe corpus, authored through a standardized editorial process that decomposes instructions into discrete steps, each paired with corresponding images. We design a novel set of guidelines and procedures to collect TEO annotations as graphs via crowdsourcing, which can be extended to new recipes, languages, and domains. Using MATEO, we evaluate six state-of-the-art LVLMs across model scales, varying language context, multimodal input structure, and fine-tuning strategies, showing the importance of multimodal inputs as well as limitations of these models.

Does a lexical retriever suffice as large language models (LLMs) become more capable within an agentic search loop? We revisit this question by pairing BM25 with increasingly capable LLMs and introduce Piika, a search agent equipped with three tools for retrieving, browsing, and reading documents. On the BrowseComp-Plus benchmark, under the same GPT-5 + BM25 setup, Piika improves the accuracy from 58.3% to 74.6% while reducing cost from $400 to $95. Despite relying only on lexical retrieval, it also outperforms the previously reported GPT-5 + dense-retriever setup, which achieves 73.0% accuracy at $361. Piika further scales with stronger LLMs, reaching 83.1% answer accuracy and 94.7% surfaced-evidence recall with GPT-5.5. Across model families, our results show that simple lexical retrieval, coupled with capable LLMs, a lightweight harness, and an effective tool interface, can remain highly competitive with more complex retrieval systems. Source code and evaluation artifacts are available at https://piika.ai.

The integration of large language models (LLMs) into video generation has enabled rapid text-to-video creation and improved visual quality. However, it still falls short of professional filmmaking, where cinematographic language is less refined than human-crafted camera work and multi-shot continuity remains challenging. To address these limitations, we introduce CamPilot, a multi-agent framework that integrates cinematographic planning and camera-work control to produce more coherent, logically structured, and human-aesthetic movies. CamPilot adopts a GRPO-based learning paradigm to learn camera work planning from 14K real-world professional movies, internalizing motion patterns and composition principles that support reasoning over shooting techniques (e.g., camera angle, motion, and focal behavior) and cross-shot relationships for controllable camera-viewpoint generation. Multiple agents further collaborate and evolve to improve overall output quality. To support this work and further studies in this domain, we establish CamEval, a benchmark for evaluating camera work quality and cinematic engagement. Empirical results show that CamPilot outperforms state-of-the-art text-to-movie generation methods on cinematographic control and quality, highlighting the impact of professional camera design on movie generation.

Two conversations can end with equally correct answers while requiring unequal user effort. One system may correct its own prior error. Another may succeed only after the user restores an instruction, referent, or evidential standard. Output-focused evaluation can assign these trajectories the same result and overlook who performed the work required for recovery. We propose a framework for measuring this difference through observable repair sequences in conversations between humans and large language models. Extending Pickering and Garrod’s Interactive Alignment Model, the framework combines four concurrent functional loops (Grounding, Continuity, Negotiation, and Epistemic) with a structural classification of who initiates and resolves repair and a temporal hypothesis about expectation stabilization. It generates testable predictions about system self-repair, epistemic signaling, and phase-dependent repair cost. In a pilot analysis of 30 dialogues, two human annotators achieved substantial agreement under an earlier turn-level rubric (Cohen’s κ = .75). The analysis revealed category-boundary problems that motivate the revised sequence-level framework presented here. Its evidential scope is limited to annotation feasibility. The resulting account makes repair burden measurable and specifies the evidence required for future validation.