For the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026), a dedicated Industry Track was organised to provide a forum for work on real-world applications of speech and language technologies. The track received 931 submissions, which were evaluated by 733 reviewers and 158 area chairs. Following the review process, 265 papers were selected for presentation, including 23 oral presentations and 242 posters. Across the accepted programme, prominent themes include evaluation and benchmarking, agents and orchestration, efficient serving and cost control, retrieval and knowledge grounding, and safety and reliability. This overview summarises the scope of the track and its submission and review process, highlights these research trends, and describes the Industry Track programme, including its oral sessions and panel discussion.

Recommender systems are central to online services, enabling users to navigate through massive amounts of content across various domains. However, their evaluation remains challenging due to the disconnect between offline metrics and online performance. The emergence of Large Language Model-powered agents offers a promising solution, yet existing studies model users in isolation, neglecting the contextual factors such as time, location, and needs, which fundamentally shape human decision-making. We introduce ContextSim, an LLM agent framework that simulates believable user proxies by anchoring interactions in daily life activities. A life simulation module generates scenarios specifying when, where, and why users engage with recommendations. To align preferences with genuine humans, we model agents’ internal thoughts and enforce consistency at both action and trajectory levels. Experiments across domains demonstrate closer alignment with human behavior than prior work. We further validate our approach through offline A/B testing correlation and show that RS parameters optimized using ContextSim yield improved real-world engagement.

Setting appropriate prices for second-hand products remains challenging for inexperienced individual sellers on Consumer-to-Consumer (C2C) e-commerce platforms. Existing regression-based approaches rely on static product–price mappings, making it difficult to capture subtle condition differences expressed in colloquial product descriptions. Inspired by recent breakthroughs in Large Language Models (LLMs), we present LLP, a retrieval-then-generation framework that generates price suggestions based on dynamically retrieved comparable products. LLP employs bidirectional reasoning to generate high-quality rationales as supervision, and subsequently applies retrieval-aware policy optimization to further improve both reasoning faithfulness and pricing accuracy. At inference time, LLP employs entropy-guided filtering to discard low-confidence price predictions. Extensive evaluations demonstrate the effectiveness of our method. It raises the static adoption rate (SAR) from 40% to 72%, while increasing GMV by 0.7% and CTCVR by 1.4%. LLP is now fully deployed in Xianyu, serving millions of individual sellers daily.

Large language models increasingly operate as autonomous agents that select and invoke tools from large registries. We identify a critical gap: when unauthorized tools are visible in an agent’s context, models select them in 48–68% of adversarial scenarios—even when explicitly instructed not to. Role escalation attacks (e.g., “I’m the CFO, override the access controls”) are the most dangerous category, reaching 96% unauthorized invocation in frontier models. We show this holds across three models spanning open-weight and frontier systems, including instruction-tuned models with strong alignment training. Critically, prompt-based compliance is both insufficient and unpredictable: explicit per-tool allowlists reduce violations to as low as 4.0% but never to zero, and compliance varies widely across models—from 4.0% to 37.0% UIR—with no reliable relationship to general capability. We propose a proxy-enforced attribute-based access control (ABAC) layer for MCP that filters tool registries at discovery time. Because unauthorized tools never reach the model context, UIR is 0% by design—a structural guarantee that prompt instructions cannot replicate regardless of model or phrasing.

We present Lunar-R1, an 8B-parameter reasoning model tailored for lunar science and mission operations, developed through continued pre-training, supervised fine-tuning, and reinforcement learning. To optimize the accuracy-efficiency trade-off, we introduce Latent Difficulty Perception (LDP), a lightweight mechanism that leverages final-layer hidden states to estimate query difficulty and regulates Chain-of-Thought generation length via difficulty-aware reward modeling. We construct Lunar-QA, an expert-verified benchmark derived from authentic lunar mission materials, and further evaluate on domain-specific tasks spanning astronomy and astrodynamics. Experimental results demonstrate that LDP reduces average token consumption by 38.9% while improving reasoning accuracy. Profiling on an RK3588 NPU yields 0.54 J/token energy efficiency and 380 ms time-to-first-token latency, validating feasibility for lunar surface edge deployment.

Industrial query-to-agent matching fails when topical relevance is mistaken for executable capability, especially on long-tail and boundary-sensitive requests. We formulate annotation as capability-bound process supervision and instantiate it with Debate-to-Skill, which uses reusable decision principles, structured deliberation, verifier-based verdict extraction, and disagreement-driven refinement. On an industrial Query2Agent benchmark, we compare Debate-to-Skill with direct-label supervision, reasoning-SFT, and structural ablations. The results test whether gains come from supervising the capability-critical decision process itself, especially on grey-zone cases where semantic relatedness and executable capability diverge.

Early in deployment, an AI search system typically operates over a frequently updated product catalog, so the available items and their properties cannot be treated as stable knowledge that can be encoded in fixed prompts or strategies. Fine-tuning, reinforcement learning, and static prompt patches fit poorly: labels are scarce, rewards drift with inventory, model releases are costly, and prompt fixes quickly stale. We present Inventory-Grounded Policy-Level Optimization (IGPO), a training-free approach for fixed AI search pipelines. IGPO separates policy from environment facts: it learns Policy Guidelines for acting on runtime inventory evidence rather than memorizing available materials. Online, IGPO grounds each query by probing the inventory and constructing an inventory portrait, then injects relevant Policy Guidelines into the retrieval and selection prompts. Offline, stochastic rollouts are grouped by query—mixed outcome groups directly yield contrastive signal, and an inventory-guided exploration loop distinguishes missed retrieval routes from cases where no matching support is found under the observed inventory evidence. Since May 2026, IGPO has been deployed in a commercial smart-assistant AI search system. A 14-day online A/B test of the complete IGPO treatment shows a 3.17% relative CTR lift and a 38.9% reduction in audited bad cases.

As large language models (LLMs) increasingly assist scientific writing, the limitations and token costs of generating TeX become increasingly visible. This paper analyzes TeX’s architectural mismatch with LLM workflows, stemming from its lack of an explicit structural representation, to illustrate its limitations on generated semantics and error localization. As an alternative, we introduce .tmu, a low-entropy tree-structured representation. With its efficient data structure and clear contextual boundaries, .tmu outperforms .tex in the above aspects. Experiments across four LLMs provide evidence for this claim in most evaluated settings. Furthermore, we show that due to its lower information entropy, fine-tuning LLMs on .tmu achieves approximately 43% lower final training loss than on .tex. Our work provides a more scalable and LLM-friendly data representation for LLM-assisted scientific writing.

Existing LLM data-analyst benchmarks fix data access at full disclosure and evaluate only on answer accuracy. But real deployments rarely grant full access. No prior work, to our knowledge, asks *"what’s the least amount of information needed for the agent to work?"*. In this article, we treat dataset disclosure as a controlled treatment along a *graduated information-disclosure* (GID) axis: C₀ (column names) and C∞ (full table) bracket a nested chain C₁ ⊆ … ⊆ C₅ that adds per-column metadata, dataset purpose, summary statistics, sampled cells, and sampled rows. We maintain agent architecture, tool registry, and decoding fixed, and score the final reports on a 14-attribute rubric across a 6 × 4 × 12 panel of frontier models, planted-signal datasets, and disclosure levels. At a high level, from C₀ to C∞, seven attributes rise and five fall. However, the rise/fall pattern has nuance, the optimal intermediate level depends on several factors including models, datasets and attributes. Level C₁ - one sentence of types and units beyond column names - matches (statistically) C∞ on signal recovery and decoy rejection at ∼49× lower prompt-token cost. The key takeaway from this work is - **current LLM data-science agents can act on partial information alone**, so row-level records need never leave the institution. Hence, the current framework also offers a way to enforce data boundary constraints (required by GDPR) without compromising on quality.

Cloud incident response increasingly relies on LLM agents that translate heterogeneous telemetry and natural-language alerts into operator-facing diagnoses. However, in production RCA workflows, LLM agents commonly serialize these structured signals into text, losing the temporal and topological structure that is critical for diagnosis. We present Knsight, an industry-deployed RCA agent that converts operational telemetry into chart-image canvases and uses a frozen multimodal LLM to reason directly over visual evidence. Knsight introduces Vision Recursive-of-Thought (V-RoT), an iterative diagnose-review-refine loop in which a Reviewer module inspects each canvas and decides whether to refine, zoom in, or stop. To adapt across recurring production failures, Knsight further converts Reviewer feedback into a shared, alias-substituted skill library used at inference time. Offline, on the OpenRCA benchmark, Knsight improves aggregate Correct by 7.0 points over the strongest text-only baseline and reduces timeouts from 117 to 11. Online, in a 90-day deployment across 1,284 production incidents, Knsight increases top-1 acceptance from 38.9% to 71.4% and reduces time-to-acknowledge and time-to-resolve by 59% and 32%, respectively.

Chinese query correction (CQC) is important for search and query recommendation on content platforms, but supervised methods rely on large annotated correction pairs that are costly to maintain as query vocabularies evolve. Unsupervised correction with language models is attractive, yet in the short-query setting, unconstrained generation often over-corrects ambiguous inputs toward high-frequency phrases, causing intent drift. We propose GUIDE, a generative unsupervised framework for CQC based on a confuse-then-clarify paradigm. GUIDE encodes phonetically or visually confusable characters with shared-IDs and reconstructs the original query with an encoder–decoder architecture, which constrains correction to plausible confusion neighborhoods while learning from unlabeled query streams. A time-decayed, query-frequency-weighted objective further supports adaptation to rapidly changing query vocabularies. Experiments on QSpell 250K and KwaiSearch, a large-scale industrial dataset from Kuaishou, show that GUIDE consistently outperforms strong baselines, while online A/B testing further confirms gains in correction quality and downstream engagement.

Small language models are attractive in production because they are inexpensive, fast, and easy to specialize, but specialization remains a difficult loop. The hard decisions surround training: acquiring and curating supervision, diagnosing failures, selecting a training strategy, and preventing a targeted fix from breaking behavior that already works. We present Pioneer Agent, a closed-loop system that automates this lifecycle. In cold-start mode, the agent turns a task specification into a trained model by searching jointly over data composition, hyperparameters, and learning strategy. In production mode, it converts judged failures into a failure taxonomy and corrective curriculum, retrains, and retains an update only when it passes regression gates. We introduce AdaptFT-Bench, seven staged scenarios that combine fixable noise with harmful supervision at increasing poison rates. Across eight cold-start benchmarks, Pioneer Agent improves over the base checkpoint on every task. Across all seven AdaptFT-Bench scenarios, it improves or preserves the deployed checkpoint while indiscriminate retraining degrades, with final gaps up to 43 points. In two production-style case studies, the same loop raises intent-classification pass rate from 84.9% to 99.3% and entity F1 from 0.345 to 0.810. These results show that deployment failures can be converted into effective training curricula without manual intervention between iterations.

Estimation of product procurement cost is a key capability for price governance on large-scale e-commerce platforms: from images, structured attributes, comparable products, and pricing knowledge, a practical system must infer a reasonable cost to surface suspiciously inflated or potentially non-compliant listings. Directly invoking general-purpose VLMs is prohibitive at high QPS (queries per second), and their verbose rationales complicate auditing and rule integration. We present Cost-LLM, a compact 7B multimodal cost-estimation model trained by a two-stage post-training pipeline: segment-normalized SFT on progressive pricing rationales, followed by GRPO with an asymmetric MAPE-based reward aligned with cost accuracy. On an in-house benchmark covering 105 fine-grained categories, Cost-LLM reaches MAPE 29.47% and Acc@20 40.42%, yielding the best point estimates among the evaluated systems. Deployed as an auxiliary signal in a price-anomaly pipeline, it provides a 30.73% relative complementary-detection yield over the production baseline at an audited precision-matched operating point. Its locally deployed serving latency is 0.4-0.8 s per listing, compared with 3-8 s observed for external commercial LLM/VLM APIs. We share the design choices, training recipe, and deployment lessons.

In local-life service platforms, query suggestion reduces user effort by generating candidate queries from input prefixes. Traditional multi-stage systems rely heavily on historical popular queries, limiting their ability to capture long-tail and emerging demand. Although LLMs provide strong semantic generalization, their deployment in local-life services faces three challenges: insufficient city-preference awareness, exposure bias in preference optimization, and strict online latency constraints. We propose LocalSUG, an LLM-based query suggestion framework for local-life services. LocalSUG mines city-preference-enhanced candidates from term co-occurrence and injects them into prompts as dynamic references rather than fusing them into model parameters. This allows the model to adapt to changing city preferences, such as merchant openings or closures, while reducing stale or locally invalid suggestions. We further introduce a beam-search-driven GRPO algorithm to align training with inference-time decoding and optimize relevance together with business-oriented rewards. Finally, quality-aware beam acceleration and vocabulary pruning reduce online latency while preserving generation quality. Offline evaluations and large-scale online A/B testing show that LocalSUG improves CTR by +0.35% and reduces the low/no-result rate by 3.98%, demonstrating its effectiveness in real-world deployment.

Production multimodal systems must serve complex vision-language tasks under strict latency, memory, and cost constraints. Frontier multimodal large language models (MLLMs) deliver strong reasoning but are prohibitively expensive for online serving; compact alternatives fit deployment budgets but lack structured reasoning ability. We present OmniThoughtVis, a scalable distillation pipeline that converts open-source multimodal corpora into deployment-ready chain-of-thought (CoT) supervision for compact MLLMs. Starting from a diverse seed pool, the pipeline generates structured reasoning traces from a high-capacity teacher, jointly annotates reasoning difficulty, answer quality, and semantic task tags, and applies difficulty-aware filtering with tag-based diversity sampling to produce a reusable 1.8M-sample candidate pool. Practitioners construct budget-specific training subsets from this pool without re-running distillation. A key practical finding is that reasoning difficulty is a far more effective data selection signal than answer quality, which becomes uninformative when teacher outputs are already strong. Distilled Qwen3-VL models (2B-8B) consistently improve over undistilled baselines across nine benchmarks, with the distilled 4B model matching or exceeding the undistilled 8B on multiple tasks. In production, replacing a large-model serving pipeline with the distilled compact model reduces end-to-end response latency by 79% while maintaining comparable downstream quality.

SQL code completion boosts developer productivity by predicting missing snippets from editing SQL contexts. However, existing solutions yield low acceptance since they cannot abstain when no suggestion is needed and lack SQL-specific training. To address both limitations, we introduce PACE-SQL, a preference-aligned framework that improves SQL completion accuracy and user preference alignment. We build two datasets with completion and abstention samples: WeSQL-1.5M from real-world interaction logs to capture authentic user preferences, and OpenSQL-FIM from public SQL corpora. We adopt continual pretraining and supervised fine-tuning to improve completion quality, and design a reward function in the reinforcement learning to enhance alignment with user preferences. Experiments show PACE-SQL improves Exact Match and Edit Similarity scores by 9.67% and 7.52% over the best baseline on WeSQL-1.5M and raises online user acceptance rate from 5% to 29%.

Large-scale content moderation requires models that can apply complex, evolving policies under limited expert supervision. We present an empirical study of reinforcement learning (RL) for policy-grounded moderation across three production-derived tasks. Using established RL and inference primitives, we develop three integrity-specific designs: a policy-grounded full-trace reward, rationale-marginalized Monte Carlo scoring, and policy-structured reflection. Each design addresses an observed failure mode, including reasoning-length collapse and polarized confidence estimates. Across the three tasks, RL-Only systems trained on 200–836 labeled examples remain competitive with supervised fine-tuning (SFT) systems trained on thousands of examples, corresponding to up to approximately one order of magnitude greater label efficiency at the evaluated operating points. Performance generally improves with additional data and training compute before saturating, although larger rollout counts exhibit variability. The strongest SFT→RL systems attain point-estimate performance comparable to policy-trained full-time employee (FTE) reviewers across the three tasks.

Production text-to-SQL pipelines often end with an LLM-as-judge that nobody has checked against humans. We audited ours and found it broken: the deployed gpt-4o-mini judge agreed with two-author gold at Cohen’s κ = 0.04 on a disagreement-enriched set and 0.42 on a uniform-random spot-check, with most over-flags traceable to a single mechanism we call Grade-Hallucination. Replacing it with self-hosted Qwen3.6-27B (κ = 0.72) reaches the same range as Claude Opus 4.7 (κ = 0.71); the comparison is underpowered at n = 96, but the deployment decision is driven by Qwen costing about 1/300 per call. Ensembling is not a free win: pairing the weak judge with a stronger one degrades agreement; three strong judges with unanimity routing reach κ = 0.79 at 89.7% auto-coverage. Applying the audit recipe out-of-domain flags 25.5% of BIRD-financial’s expert-authored gold SQLs as likely substantive issues under our annotation protocol. Code and pre-registration are at https://anonymous.4open.science/r/synca-audit-CBE1; the datasets are deferred to the post-publication release for anonymization reasons.

Tool-augmented LLM agents discover relevant APIs from large registries before planning execution, yet finite context windows prohibit injecting all tool definitions into a single prompt. The dominant approach, which we term Retrieve-then-Plan (RtP), embeds the user query and retrieves similar tools via approximate nearest-neighbor search. We study an alternative Plan-then-Retrieve (PtR) paradigm in which the LLM first generates Abstract Tool Stubs (hypothesized function signatures describing needed capabilities) which then serve as retrieval queries against the registry. Evaluating on ToolBench (46,984 unique API endpoints, 50 categories) across single-tool (G1), intra-category multi-tool (G2), and cross-category multi-tool (G3) query subsets, we find a clear complexity-dependent crossover: RtP outperforms PtR on single-tool queries while PtR outperforms RtP on multi-tool queries. This crossover holds at every retrieval budget (k=1 to 20), is robust across registry sizes from 1k to 47k APIs, and is consistent across model families (Gemini 2.5 Flash and gpt-oss-120b). We characterize a stub-hallucination failure mode (the LLM invents plausible but non-existent API names) that acts as a double-edged sword: on simple queries PtR’s stubs abstract away proper-noun signals that RtP matches directly, yet on complex queries the same mechanism decomposes multi-faceted intent into functional sub-needs. We further analyze token overhead and show that PtR’s extra discovery call roughly doubles inference cost, which pays off on multi-tool queries but hurts on single-tool ones, motivating complexity-adaptive routing as a practical direction — one that can be driven by a zero-cost signal already present in the pipeline (RtP’s top-1 retrieval score).

As LLM agents are exposed to hundreds to tens of thousands of skills, tools, and API functions, full-library prompting becomes costly, slow, and less reliable: each added candidate increases prompt tokens and latency, while longer candidate lists introduce more distractors for LLM selection. We present Toollery, a training-free candidate-compression framework for scalable LLM skill/tool selection. Following established document-side query expansion, Toollery generates user-intent queries from each skill/tool specification and builds a retrieval index that maps real user requests to compact candidate sets before final LLM decision-making. By treating high-level skills and atomic tools as selectable capabilities, Toollery can be applied to both skill libraries and tool registries. We evaluate Toollery on the roughly 79K-capability SkillRouter benchmark, BFCL-V4 with over 440 atomic tools, and 3,396 proprietary smart-cockpit requests over 220 tools. Across these settings, Toollery keeps online selection bounded to a compact top-k candidate set and improves recall over ordinary specification retrieval. At a fixed top-10 budget, Toollery improves end-to-end selection on the cockpit dataset, and maintains comparable AST Accuracy on BFCL-V4. These results support Toollery as a practical candidate-compression framework for large and evolving agent capability libraries, while showing that quality and cost gains depend on workload coverage and provider caching.

Effective troubleshooting agents in enterprise customer support depend on retrieving actionable guidance from similar historical resolved cases, yet existing retrieval-augmented generation (RAG) systems treat support cases as static documents and overlook their multi-stage, stateful nature. We introduce RAFT (Retrieval-Augmented Framework for Troubleshooting Agents), a stateful RAG framework that abstracts each resolved case into a directed chain of timeline entries and retrieves at the entry level, surfacing cases whose intermediate states match the active case and returning the parent-case trajectory anchored at the matched state; an optional case-level graph links cases by shared root cause and resolution. We evaluate this retrieval layer directly, which, unlike evaluating a full agent system, requires no production deployment. Because public multi-stage troubleshooting data is extremely rare, we pair a synthetic benchmark built from Microsoft Learn Windows Server documentation with real Apache Jira issues carrying human-created duplicate labels. RAFT improves Case Hit over vanilla RAG and GraphRAG baselines at every stage of case progress, with statistically significant gains over the strongest baseline; the Jira results provide directional evidence that the advantage transfers to real case histories. We release our benchmark, implementation, and the Apache Jira evaluation set.

User interface (UI) and user experience (UX) evaluation is central to product development, yet reliable feedback still relies on recruiting human participants or running online A/B tests, making early-stage iteration slow and costly. In light of this, recent work has explored Multimodal Large Language Models as proxy evaluators. However, existing approaches either produce surface-level critiques or a judgment that reflects the model’s own biases rather than the genuine response of a particular user. We introduce PerceptUI, a framework for persona-conditioned UI/UX evaluation that predicts how a specific user would answer interface-related questions and produces natural-language rationales. PerceptUI is trained in two stages: (i) contrastive reflection fine-tuning distills teacher-generated rationales by extracting lessons from human decisions, and (ii) reflective prompt-evolution, which refines prompts using the model’s failure traces. Across multiple domains and datasets, PerceptUI improves alignment with human responses, generalizes to unseen questions and personas, and yields population-level response distributions.

Identifying proxy issuers and bonds is a critical but largely manual step in illiquid corporate bond pricing, requiring analysts to navigate sparse public information and inconsistent industry signals. We present a modular pipeline for proxy retrieval that integrates issuer profiling with hierarchical industry classification and multi-stage retrieval over issuers. We evaluate alternative design choices across hierarchical classification and reranking stages to understand how structured representations and industry-aware signals influence performance. Experiments on a large internal dataset show that domain-adapted embeddings and explicit industry hierarchy signals are primary drivers of quality across stages. The evaluated system produces issuer-level proxy shortlists rather than final bond prices. Pilot deployment experience highlights the value of modular, cost-aware designs for robust proxy retrieval in enterprise settings.

LLMs can personalize education, although current static-prompt tutoring systems struggle to adapt to diverse academic disciplines. We develop and test a system with subject-aware prompting, based on 14 pedagogical features (e.g., tutor scaffolding, student understanding) extracted from raw transcripts. We first train a prompt routing model in a simulation environment, and then deploy it for online adaptation with actual high-school students. The simulation benchmark shows the router outperforming two static baselines (0.694 vs. 0.647 and 0.64, p<0.001). A/B testing (N=656 conversations from 359 students) shows sim-to-real transfer where the model switches from analytical to scaffolding learning strategies. Our adaptive prompt selection mechanism improves instructional efficiency, maintains pedagogical quality and reduces interactions by around 3 turns (p=0.007). While a greedy router achieves a comparable exercise conversion rate with the baseline (19.1% vs. 19.6%), a stochastic router that samples strategies leads to a higher conversion rate (28.1%).

Multi-agent LLM systems that generate structured workflows from natural-language requests are now deployed in production across cloud automation, DevOps, and enterprise orchestration. Operating them exposes a recurring change-management problem. Routine updates, such as re-running an input, swapping the LLM, or refactoring an agent’s prompt or orchestration code, often produce workflows that differ substantially from validated references. Engineers then lack a principled way to decide whether a change is safe to ship. Automatic workflow evaluation is the natural tool, but in practice metric scores are poorly calibrated, and a numeric change rarely communicates the severity of the underlying degradation. We introduce WorkflowPerturb, a controlled benchmark that applies realistic, graded perturbations to golden workflows. It contains 4,973 golden workflows and 44,757 perturbed variants across three perturbation types (Missing Steps, Compressed Steps, Description Changes) at severities of 10%, 30%, and 50%. We benchmark multiple metric families, analyzing their sensitivity and calibration using expected score trajectories and instance-level alert rates. Our results characterize systematic differences across families and support severity-aware interpretation of workflow evaluation scores in change-management settings. WorkflowPerturb is publicly available at https://huggingface.co/datasets/microsoft/WorkflowPerturb.

Target Safety Assessment (TSA) requires systematic integration of genetic, transcriptomic, target homology, pharmacological, and clinical data to evaluate potential safety liabilities of therapeutic targets. This process is labour-intensive and expert-dependent, posing challenges in scalability and reproducibility. We present TSAssistant, a human-in-the-loop multi-agent framework that decomposes TSA report generation into a workflow of specialised subagents, each responsible for a single report section. Each subagent retrieves evidence from curated biomedical sources through standardised tool interfaces and produces individually citable, evidence-grounded sections, with behaviour shaped by a hierarchical instruction architecture that separates coordination logic from domain expertise and user intent. To complement these soft constraints, programmatic execution hooks and persistent memory stores enforce hard constraints across the workflow, while an interactive refinement loop allows experts to review and revise individual sections with full conversational context preserved across iterations.

Large-scale urban simulation plays a pivotal role in social science, traffic safety, and transportation policy. Recent work has shown that large language models, when prompted as agents, can generate lifelike daily routines at city scale. Yet these methods typically rely on few-shot prompting, causing agents to reproduce the LLM’s behavioral priors rather than the target population. We introduce CityReal, a modular framework for human-aligned urban simulation. CityReal models agents as intention-driven decision makers that pursue coherent mobility and activity plans rather than isolated step-by-step choices. They adapt over time by learning habits and preferences based on experience and constraints. To improve population-level realism, we learn textual adapters for behavior modules that align agent decisions with observed population statistics. Experiments show that CityReal improves alignment with real-world human behavior at both micro and macro levels. Scaling to tens of thousands of agents, it supports analysis of crowd density, place popularity, mobility flows, and well-being under different urban scenarios, offering a scalable testbed for urban simulation and forecasting.

As agents grow more capable, legal-domain LLM agents promise to turn document-heavy matters into reviewable work products—yet reliable deployment faces three obstacles: no large-scale evidence on how today’s strongest model-and-harness combinations behave on end-to-end legal matters; no agent architecture adapted to the legal vertical, only general-purpose harnesses; and, in a setting that keeps shifting with new facts, authorities, and deadlines, no mechanism for systems to learn from their own outcomes. We address each. A large-scale empirical study on Harvey LAB—12,510 agent trajectories—shows that even frontier agents remain far from completing matters in a single pass: per-criterion accuracy climbs with stronger models while strict matter completion stalls. We then introduce Parthenon, a self-evolving legal-agent framework that factors Model, Harness, Agent roles, legal Knowledge, deterministic Tools, and procedural Skills into auditable surfaces for source traceability, date and number grounding, deliverable compliance, and issue closure. Finally, an anti-leakage learning loop converts scored failures into task-agnostic edits to skills, tools, and knowledge, letting the system improve with experience—as a firm refines its checklists and playbooks after each matter—without touching model weights. Across our large-scale empirical analysis, Parthenon substantially improves the performance of state-of-the-art models and harnesses on legal-matter tasks.

Translating sequential programming priors into the parallel temporal logic of hardware design remains a crucial bottleneck for large language models (LLMs). To investigate this, we introduce an operational error taxonomy grounded in the Register-Transfer Level (RTL) coding pipeline. Our taxonomy categorizes failures into extraction/syntax (L1), compilation/elaboration (L2), and testbench-observed functional (L3) stages under a fixed evaluator. Evaluations reveal an empirical ceiling on the VerilogEval benchmark, as frontier models plateau at a 90.8% initial pass rate. Furthermore, we expose a striking surface convergence gap: optimization readily eliminates syntax errors but concurrently exacerbates deeper functional failures. Our findings demonstrate that alignment techniques merely teach models to compile. While test-time scaling can patch solvable errors, RTL coding capacity remains bounded by pretraining knowledge. Addressing challenges in the current LLM-based hardware generation pipeline requires more studies in model reasoning rather than alignment interventions.

Fine-tuning large language models (LLMs) for e-commerce attribute extraction requires labeled data representative across thousands of product types, attributes, and multiple languages. This combinatorial scale translates to millions of annotations, rendering human labeling prohibitively costly. While recent work has demonstrated synthetic label generation using LLMs, deploying such approaches at industrial scale requires integrated quality control mechanisms. We present SynthAVE, a large-scale benchmark for attribute-value verification spanning 12,726 products across 229 product types, 792 attributes, and 4 languages (Spanish, French, Italian, German). To validate synthetic labels at scale, we introduce a multi-LLM arena framework where each sample is evaluated by 21 judge configurations (7 model families × 3 prompts), with final labels determined via majority voting; disagreements with the synthetic label are expert-adjudicated and agreement cases are audited on a stratified sample. The majority vote ensemble agrees with expert labels at Cohen’s 𝜅 = 0.92 (95.0% agreement), while individual judges agree with one another only moderately (Fleiss’ 𝜅 = 0.76)—by design, since we select for diversity. This demonstrates that diverse models with varying individual judgments aggregate into highly reliable predictions, enabling cost-effective validation at scale while concentrating expert effort on the cases where it changes the label. We estimate the resulting label quality at 97.9%.

The dominant approach to building domain-specific pretraining corpora is to filter large web archives such as CommonCrawl. This works well for popular domains but breaks down for specialized ones, where relevant content is sparse and often beyond the reach of popularity-driven crawlers. We present **Data Scout**, which inverts this: instead of filtering an archive, it directs a targeted crawl. An LLM expands a root topic into a taxonomy and thousands of search queries; the returned URLs (*seeds*) are grouped by subdomain and screened with a user-supplied classifier (the *probe*), admitting each subdomain on the basis of a small sample. This works because relevance has a sharp boundary at the subdomain level: in mathematics, a page is 21x more likely to be relevant than one on a sibling subdomain. With the FineMath classifier as the probe, **21.9%** of crawled pages are high-quality math content, **70x** the 0.31% rate from filtering a comparable web sample, so the crawl wastes far less effort. But the payoff is not just efficiency: 63.2% of these pages are missing from CommonCrawl altogether, yet just as useful for training. Continued pretraining of Llama-3.2-3B on 1.9B Data Scout tokens matches FineMath corpus on GSM8k. Because the probe is the only domain-specific component, Data Scout can in principle apply to any domain with such a classifier.

This paper focuses on automatically generating informative ad descriptions in sponsored search. Unlike ad titles which are usually optimized to attract user click feedbacks, ad descriptions have a longer text span and possess the potential of incorporating world knowledge to address user search intents while presenting the fine-grained selling points of the ads. We propose Interactor, a multi-turn iterative creation framework optimized with agentic RL for ad description generation. The generation model acts as a policy that interacts with a customized environment consisting of multiple generative reward models. Given initial generations by the policy, the customized GenRMs evaluate qualities including knowledge capacity and landing page consistency, providing both binary signals and detailed feedbacks. The policy then iteratively refines the descriptions based on such feedbacks to ensure continuous improvement. Experiments show that it significantly outperforms state-of-the-art ad text generation approaches in generating knowledge-rich and faithful ad descriptions. Since late May 2026, it has been deployed online in a leading search ads system, where the framework serves over 140k advertisers, contributing to both ad revenue and user experience.

Traditional intent classifiers map utterances directly to intent categories using labeled examples. In contrast, LLM-assisted zero-shot intent classifiers rely on natural language descriptions alongside intent labels to determine the most appropriate categories for a given utterance. As intent taxonomies grow and evolve, descriptions must define clear boundaries among overlapping intent categories. Maintaining them is costly because it requires domain expertise and continual updates to keep pace with changing product and user behavior. We present MOSAIC (Multi-label Ontology Synthesis for Automated Intent Classification), a deployed system that automatically generates and updates intent descriptions for an existing taxonomy. MOSAIC analyzes unlabeled user utterances to identify coherent clusters. It associates cluster-level evidence with the existing intent taxonomy. Then, it synthesizes discriminative descriptions that define each category’s semantic boundary. On a 140-class production deployment, MOSAIC approaches the quality of expert-written descriptions while reducing taxonomy revision time from 2-4 weeks to less than 30 minutes. On the public NLU++ and Multi³NLU++ benchmark datasets, MOSAIC-generated descriptions do match or exceed the performance of expert-written descriptions.

Retrieval-Augmented Generation (RAG) systems are vulnerable to corpus poisoning attacks that manipulate model outputs through malicious retrieved documents. Existing detection methods typically rely on auxiliary classifiers or additional LLM-based verification, introducing substantial computational overhead. We present TRACE, a lightweight detection framework that identifies poisoning attacks by tracing answer-related tokens through token influence attribution. TRACE first discovers recurrent high-influence keywords across retrieved documents and then performs a secondary verification to confirm their influence on model predictions. Experiments on three QA benchmarks and six LLMs demonstrate strong detection performance while simultaneously uncovering attacker-specified target answers.

Time series (TS) reasoning models (TSRMs) have shown promising capabilities in general domains, yet they consistently fail on financial domain, which exhibit unique characteristics. We propose a general 2 x 2 capability taxonomy for TSRMs by crossing 1) single-entity vs. multi-entity analysis with 2) assessment of the current state vs. prediction of future behavior. We instantiate this taxonomy in the financial domain—where the distinction between deterministic assessment and stochastic prediction is particularly critical—as ten financial reasoning tasks, forming the FinTSRBench benchmark based on S P stocks. To this end, we propose FinSTaR (Financial Time Series Thinking and Reasoning), trained on FinTSR-Bench with distinct chain-of-thought (CoT) strategies tailored to each category. For assessment, which is deterministic, we employ Compute-in-CoT, a programmatic CoT that enables models to derive answers directly from raw prices. For prediction, which is inherently stochastic, we adopt Scenario-Aware CoT, which generates diverse scenarios before making a judgment, mirroring how financial analysts reason under uncertainty. FinSTaR achieves 78.9% average accuracy on FinTSRBench, substantially outperforming LLM and TSRM baselines. Furthermore, we show that the four capability categories are complementary and mutually reinforcing through joint training. Code is publicly available at: https://github.com/seunghan96/FinSTaR.

Access to reliable health consultation is a critical global challenge. While large language models show promise, their practical utility is typically limited by high input barriers, lack of adaptive response strategies for complex scenarios, and insufficient real-world evaluation. To address these gaps, we present a novel health AI assistant that integrates a low-latency full-duplex voice interface with a Multi-Agent System (MAS). The full-duplex design supports natural turn-taking, significantly lowering interaction barriers and quadrupling average conversation depth from 2.62 to 10.16 turns compared to text-based baselines. Furthermore, our MAS architecture decouples clinical reasoning from general conversation, effectively managing complex queries and boosting clinical performance scores by over 30% compared to single-agent baselines. We conducted the first large-scale analysis of an LLM-based health assistant deployed to the general public (N > 5,000), demonstrating superior user engagement and providing empirical evidence on the safety and utility of deploying such systems in realistic environments.

On-device LLM deployment routinely serves GGUF-quantized models whose safety is ver- ified at full precision rather than on the llama.cpp/GGUF stack end users actually run. We stress-test nine production mod- els across four GGUF levels against seven CHATBUG chat-format attacks on 520 Ad- vBench instructions, served through the same llama.cpp binary that powers Ollama, and report three deployment-relevant findings. (1) Q2_K vulnerability concentrates in the smallest on-device models: Llama-3.2-1B reaches 75.3% ASR-R at Q2_K (+26.9 pp vs. Q8_0) while Gemma-3-4B is essentially un- affected (+3.0 pp), so model choice dominates quantization-level choice within the on-device regime. (2) The vulnerability is a property of the deployed system: a subset reproduces end- to-end through Ollama (median |∆|= 2.7 pp). (3) Keyword- and moderator-based metrics diverge in magnitude: Llama-2 Mismatch-∅ shifts ASR-R +55.8 pp but ASR-M−1.7 pp at Q2_K, while Llama-2 Overflow-FS is con- firmed unsafe by both (+16.8 pp ASR-M). We release a one-page deployment-safety checklist and GGUF-native diagnostic probes, and iden- tify single-layer probes as insufficient cross- model predictors of ASR change, motivating per-layer llama.cpp instrumentation.

Generative Retrieval (GR) formulates retrieval as a sequence-to-sequence generation task, assigning each document a document identifier (DocID) and retrieving it through autoregressive decoding, making DocID design a critical factor in retrieval quality. However, existing schemes based on discrete representation learning suffer from inherent collision issues and create a mismatch between the DocID’s encoding objective and the system’s business optimization target. To address these limitations, we propose Cluster-Ranked Identifier (CRID), which decouples DocID into semantic clustering and business-value ranking, yielding collision-free identifiers that support incremental updates via intra-cluster reranking. We further introduce an analytical framework that decomposes retrieval gains into personalized preference and statistical prior generalization, revealing how semantic cluster size governs the balance between the two components. Experiments on a Taobao e-commerce corpus of over 300M items show that CRID surpasses the strongest embedding-based retrieval baseline on top-K Hitrate, and delivers +1.06% GMV in full-traffic deployment.

Institutional investors search visually dense pitch decks, board packs, and diligence materials that change hourly near deal closing. OCR followed by figure verbalisation is costly to refresh at this scale and can lose chart detail. We present PULSAR, a production vision-first retrieval system deployed at Mubadala Investment Company. PULSAR indexes page images with a frozen ColPali-style backbone and uses a pooled two-stage late-interaction index: compact page summaries support initial retrieval, followed by exact MaxSim rescoring over a finer pooled representation. On ViDoRe V3, this design reduces median vector-search latency by 15.1 times against an unpooled configuration with less than 0.01 absolute NDCG@10 and Recall@10 loss; production median vector-search latency is 156 ms. Under concurrent load, the pooled index sustains approximately 88 times higher QPS than an unpooled index. The event-driven ingestion path is estimated to be approximately 20 times cheaper per page than the OCR+verbalisation baseline it replaced. Since March 2026, PULSAR has served 78 thousand documents and approximately 2.4 million pages across more than 3,000 deals. At the production top K, it more than doubles answer-fact recall over the OCR+verbalisation baseline.

We present a retrieval-augmented generation (RAG) system in production since April 2026 for credit policy question answering, a setting in which the corpus is revised several times per quarter: a single revision changes thresholds, narrows scopes, and restructures tables. Static test sets silently decay under this regime, and scores computed against successive corpus versions cannot separate system drift from corpus drift. We propose a longitudinal-aware evaluation framework that makes RAG quality over an evolving corpus both measurable and attributable. It combines (i) a change-type taxonomy — Stable, Evolved, Added, Removed — built on a deterministic, model-independent alignment of document subsections keyed to how each gold-supporting provision changed between versions, making explicit which scores remain comparable across releases; (ii) a versioned, class-stratified synthetic QA pipeline that keeps the test set current at revision cadence; and (iii) a statement-level LLM judge for compound policy answers; each component is calibrated against subject-matter experts. Across three corpus versions, a conventional longitudinal-unaware protocol would report a recall decline of up to 25.7 points. However, paired comparisons on answer-invariant items show no evidence of regression in the unchanged system.

Audits of generative AI (GenAI) systems often summarize behavior as a reported rate: how often the audited system complies with policy. Researchers and stakeholders use that rate to compare systems, track regressions, and gate deployment. A reported rate reflects both the system under audit and the measurement choices behind it, so a change in the rate can leave it unclear whether the system or those choices moved. We introduce ASSERT, a specification-driven measurement pipeline for GenAI audits that ties each reported rate to a written specification of the measurement choices used to produce it. ASSERT helps draft a behavioral rubric and test cases, then runs the audit against a GenAI system and returns a reported rate. In a case study on conversational deception, we observe that the reported rate moves substantially with the dialogue setup, the simulated user, the judge, and the evidence bar for non-compliance. These measurement choices substantially change the reported rate and can reorder GenAI system rankings. Because each reported rate is tied to an explicit specification, differences across audits are easier to attribute and interpret.

General-purpose Text-to-SQL systems achieve strong performance on academic benchmarks like Spider and BIRD, where schemas are relatively shallow and column values are often human readable. In production financial databases, where concepts are stored as opaque integer keys rather than human-readable strings, these methods fall below 50%, as even simple queries require multiple joins and filter predicates reference opaque IDs. We present FLINT, a domain-specialized Text-to-SQL system that closes this gap through three key components: (1) a lookup agent that dynamically resolves natural-language concepts to question-specific reference table constraints, (2) embedding-based retrieval of structurally similar query templates from a compact, expert-authored bank, and (3) schema linking that prunes a large table schema to the relevant subset by traversing foreign-key chains, rather than relying on name similarity alone. We evaluate on two datasets totaling 359 questions over production financial schemas. FLINT outperforms various state-of-the-art baselines using the same LLM. The system is deployed in production as part of a financial data retrieval service.

Software testing is essential for maintaining software reliability, yet issue reproduction tests are often missing when bugs are reported. Automatically generating such tests can accelerate debugging, support test-driven development, and improve automated software engineering systems that rely on execution-based verification. Existing approaches largely depend on proprietary language models, limiting accessibility and reproducibility. We present SWE-Tester a framework for training open-source large language models to generate issue reproduction tests from natural language issue reports. To support this task, we curate a dataset of 41K training instances from 2.6K open-source GitHub repositories. Fine-tuning on this dataset improves issue reproduction performance across multiple model families, yielding gains of up to 10% points in success rate and 21% in change coverage on SWT-Bench Verified. We further study the effects of model size, training data scale, and inference-time compute, observing consistent improvements along all three axes. Our results demonstrate that open-source models can be effectively specialized for issue reproduction test generation and provide a scalable foundation for future software testing and automated issue-resolution systems.

One model passed our fidelity check without ever opening the datasheet. We found it while qualifying models for an internal extraction service: a structured-output constraint had silently disabled tool use, and the model answered anyway, with fabricated source text. Only the per-tool trace exposed it. Fidelity—whether an extracted value matches the source—is the standard measure for agentic document extraction, and it scores that run a success. We therefore log every tool call in an agentic benchmark of 25 hand-curated claims over three components, with 12 more on a fourth, 37 in all. From that dispatch record we build two instruments: a rule-based failure- attribution classifier, and a silent-failure detector whose two rules check only which tools were called, never the extracted value. The detector raises no flag on 207 clean fidelity-passing extractions across three model families, and recovers all 50 planted faults that withhold exactly the tools its rules check. The two results are not symmetric: the first bounds the false-positive rate, the second is recall by construction, and detection power against runs that call their tools and still answer wrongly is unmeasured. A second, independent oracle, a causal chamber that tests whether the datasheet’s claims hold under physical measurement, is intentionally partial: it confirms only what the apparatus can exercise, a verifiable envelope of 2 of those 37 claims, and we give a taxonomy of why the rest are not physically gradable. Under a controlled perturbation, fidelity passes throughout while the chamber verdict flips exactly at the measurement uncertainty. Across three deployed model stacks (one destabilised by its serving stack, not by any capability gap) the tool layer buys portability and observability rather than accuracy, and earns its premium only once a document outgrows the context window. We distil the lessons into a checklist of cheap dispatch-level signals.

Existing dialogue systems rely on query suggestion to enhance user engagement. Recent approaches mainly optimize generative models using click-through rate (CTR) models to align with user preferences. However, these methods are less effective in early-stage deployment scenarios, where click feedback is sparse and insufficient for training a reliable CTR model. To bridge this gap, we propose QualEQS, a quality-first iterative reinforcement learning framework for e-commerce query suggestion. We formalize actionable suggestion quality along three dimensions that directly affect downstream usability: answerability, factuality, and information gain. To continuously improve from online traffic without click supervision, we further propose group-level disagreement among candidate suggestions to identify ambiguous query contexts and mine hard training cases for iterative refinement. We also introduce EQS-Benchmark, a dataset of 16,949 real-world e-commerce queries for offline training and evaluation. Experiments show that our quality-based offline metrics correlate strongly with online performance, providing a practical evaluation recipe for sparse-feedback deployment. In both offline and online settings, QualEQS consistently outperforms strong baselines, yielding a 6.81% improvement in online ChatPV in a real-world enterprise-level conversational shopping assistant system.

Evaluating conversational agents typically requires human participants, a process that is costly and difficult to scale. LLM-based user simulators and LLM-as-a-judge evaluators offer a promising alternative, but how close they are to replacing real users remains unclear. We study this through a human user study (52 participants) and a simulated user study (four LLM-based simulators) applied to an experimental conversational sales agent, comparing LLM-as-a-judge assessments against human self-reported ratings. The results show that none of our simulators fully replicates human interaction patterns, and none of the LLM-based evaluators shows strong correlation with human ratings on real dialogues. We provide practical guidelines for combining automated and human evaluation, identifying key areas for improvement and highlighting when real-user studies remain essential.

In the Microsoft Azure datacenter network, a technician repairs a link and an on-call engineer, on a different shift and a different continent, decides whether the repair holds. The only channel between them is the technician’s free-text note. When the note omits the measurements the engineer needs, the ticket stalls: the median blocked ticket took 7.0 days to resolve, of which roughly one hour was engineering work. The rest was waiting for an account of the work that the note did not carry.

We treat the note as the object to be repaired. A pipeline in production for ten months parses each note at write time into two lightweight schemas, STAR and ECO, using deterministic rules for measurements and a language model for prose. It reaches 0.913 F1 against a 2,141-note gold standard, above a stronger frontier model working alone, and across a staggered rollout of 450+ datacenters the deployed package cuts time-to-repair by 3.4 days. Resolution time falls steeply in a completeness score, at a rate that holds across five physically unrelated failure types, which is what we would expect if the constraint were in the handoff rather than in the physical work.

Industrial recommendation training increasingly relies on decentralized multi-GPU infrastructure, yet sparse embedding tables remain a major bottleneck due to significant storage, lookup, and communication overhead. Most existing works typically optimize only one of these bottlenecks or are designed for parameter server architectures, making them less effective in industrial decentralized training stacks. To tackle these challenges, we propose MixEmb, a practical framework for efficient decentralized embedding training through importance-aware quantization and caching. We first design a novel importance metric, i.e., embedding score, which can distinguish the contribution of embedding vectors based on spatial heterogeneity and temporal dynamics. Guided by embedding scores, MixEmb employs mixed precision quantization and hierarchical caching at fine-grained vector level. Besides, our key finding is that quantization and caching decisions are tightly coupled. Instead of naive combination, we delicately integrate them into a unified training framework through coupled co-design. A theoretical convergence analysis is also provided for MixEmb. We implement and evaluate MixEmb in real-world training scenarios. Across three workloads, MixEmb attains up to 1.55× speedup and 3.54× compression ratio compared to state-of-the-art methods.

Large-scale search systems must process vast numbers of diverse, short queries for query understanding (QU) tasks such as named entity recognition (NER) and query classification, under strict latency constraints. Yet general-purpose language models (LMs), trained largely on context-rich open-domain text, are poorly matched to context-poor queries; large generative LMs also incur high serving costs. To achieve accurate and efficient QU, we introduce SEG-QU, a session-guided continual pre-training framework that tailors standard encoder pre-training objectives to search queries. It jointly optimizes token- and query-level objectives over session-derived query pairs: Session-Guided Masked Language Modeling (SG-MLM) reduces uncertainty in single-query MLM, while Same-Session Query Prediction (SSQP) improves downstream fine-tunability. On commercial-search NER and classification tasks, SEG-QU outperforms encoder baselines and fine-tuned generative LMs up to 43× larger, while requiring substantially lower latency and energy. SEG-QU also transfers effectively to a public e-commerce benchmark. Our code is publicly available at https://github.com/lycorp-jp/segqu.

Patient contexts span hundreds of heterogeneous documents and thousands of structured data points, yet the document-level metadata that AI systems need for retrieval and triage is absent or incomplete. Standard retrieval-augmented generation fails on this data, mishandling temporal reasoning, cross-document dependencies, and missing metadata. We report on the deployment of ACIE (Agentic Clinical Information Extraction) at University Medicine Essen: an on-premise agentic RAG pipeline that reasons over complete patient contexts and grounds every answer in source passages for clinician verification. We quantify the metadata gap, trace the architectural decisions it shaped, and evaluate extraction alongside an independent retrospective lymphoma registry study, in which nuclear-medicine physicians verify every extracted value against its cited sources. Across 7,326 clinician judgments on this lymphoma-registry workflow, 96.5% of extractions were accepted, with per-type acceptance ranging from 80% to 99%.

Late-interaction retrieval (ColBERT, ColPali) scores a query against a document via the MaxSim operator. The standard PyTorch implementation materialises the full query-token × document-token similarity tensor only to reduce it away. At ColPali scale this is the single largest tensor in the pipeline (e.g. 21 GB in FP16 for 10K documents) and limits both candidate set size at inference and batch size during contrastive training. We present Flash-MaxSim (FM), an IO-aware fused GPU kernel that computes the same MaxSim scores without ever materialising the tensor, and extends the same principle to the training backward. At ColPali scale on A100, FM reduces inference peak memory by 1.4-2.6× relative to the deployed chunked baseline (4.9-8.9× relative to unchunked eager execution) and reduces MaxSim-operator training memory by two orders of magnitude, enabling exact reranking over larger resident candidate pools and contrastive batch sizes that vanilla autograd cannot fit on a single GPU. The kernel is a drop-in replacement, exact up to floating-point evaluation order under its stated FP32-accumulation protocol: nDCG@10 differs from the FP32 reference by at most 5×10^-4 on BEIR and REAL-MM-RAG. A separate INT8 path trades exactness for halved index storage at high fidelity. Released open-source at https://github.com/roipony/flash-maxsim.

Post-training is becoming a service (PTaaS): a customer hands an operator data and a goal, and a forward-deployed engineer (FDE) returns a fine-tuned, evaluated, and deployed model under a budget, a human-approval gate, and reproducibility requirements. Seating an LLM agent in the FDE seat raises a question existing benchmarks cannot answer: not whether an agent can raise a metric, but whether it can be trusted to deliver. We answer it on a governed delivery plane, where an agent drives ten stages and an oracle scores each stage from platform-recorded facts. The central silent failure is the run that trains but does not learn (TBDL): loss falls, every signal stays green, and the delivered model is no better than the base. A detector calibrated on known-corrupted runs flags every such run before payment. We ran four frontier agents (Claude Opus 5, GPT-5.6-luna, Gemini 3.7 Flash, DeepSeek-V4-Pro) end to end on metered L40S, A100, and H200 GPUs across 8B to 70B open bases, certifying every scenario before scoring. We also ran a human FDE arm under the same oracle and compare every agent against it.

Aggregating noisy, conflicting textual hypotheses into a reliable consensus is a fundamental challenge when deploying NLP systems in real-world industrial settings. While monolithic Large Language Model (LLM) agents offer unbounded expressivity for tasks like Root Cause Analysis (RCA), they suffer from context limits, compounding hallucinations, and prohibitive inference latency. Traditional weak supervision offers statistical rigor but is mathematically restricted to discrete classes. We present Loom, a generative consensus framework deployed for real-world RCA that bridges these paradigms. Loom aggregates open-form hypotheses emitted by modular heuristics (diagnostic templates dynamically populated with episode-specific entities, times, and metrics) by projecting them into a continuous embedding space, and resolves conflicting signals with an iterative centroid-based reweighting algorithm. The resulting consensus weights ground a single lightweight LLM synthesis step. Evaluated on the OpenRCA benchmark, Loom occupies the accuracy–efficiency Pareto frontier: it matches a state-of-the-art autonomous agent on Bank and Market-2 and trails on Market-1 and Telecom, while using a single LLM call per incident on all four datasets (∼26× faster; ∼33× with an 8B-parameter synthesizer). We discuss our deployment experience, highlighting lessons learned regarding the trade-offs between agentic depth and inference latency, negative results in redundancy detection, and how deterministic consensus fosters trust among Subject Matter Experts (SMEs).

B2B outbound sales involves high transaction values and relationship-based trust, so closing a deal typically requires multiple rounds of outreach building on prior context. Existing LLM-based sales agents operate within single sessions and cannot carry forward context or refine strategies from experience. Methods that do learn from experience store lessons as unstructured text and treat all outcomes as equally informative, with no mechanism to distinguish a flawed strategy from an external factor. We propose SPAR, a self-evolving framework that models selling strategies as testable hypotheses and refines them without parameter updates. A double-loop architecture lets an inner loop adapt to individual merchants through episodic memory while an outer loop evolves a shared strategy playbook through causal attribution and hypothesis-driven evolution. In a live A/B test on the 1688 platform with over 200,000 merchants, SPAR achieves +15.8% conversion lift and +35.8% high-intent merchant growth. Human evaluation by domain experts rates SPAR at 4.62 out of 5 on overall sales quality, the highest among all methods.

Deployed LLMs field hundreds of millions of conversations a day and refuse a non-trivial fraction of user requests, yet the user-side cost of refusal, whether users disengage and by how much, is unmeasured on any production model: a blind spot for the consumer-facing platforms built on them. A/B-testing refusal is ethically and operationally constrained on frontier-lab products and unavailable to third-party platforms deploying these models through a fixed API. We introduce RAW, a method that uses LLM stochasticity as a natural experiment: at near-identical prompts in any existing log, the same model sometimes refuses and sometimes does not, and that variation identifies the causal user-side effect of refusal. Applying the method to 15 deployed LLMs in LMSYS-Chat-1M and WildChat, with user-clustered inference, we find that for about half of the models we can measure precisely, a refusal costs 3 to 17 pp of subsequent re-engagement; the rest show no distinguishable effect, among them one deployment whose positive point estimate we trace to persistent users retrying the refused request. Our contribution is the measurement method, which runs on any deployed model from logs alone for about 0.30 per 1,000 conversations. We release labels, moderation scores, prompt embeddings, and pipeline code.

Extracting structured data from documents is a core task in finance, insurance, legal, and healthcare, yet empirical studies comparing extraction approaches across domains remain scarce. We assemble a corpus of 1,114 documents across 12 categories spanning 8 domains, with 7,197 fields annotated with ground-truth values, and use it to study how extraction accuracy varies across models, pipeline architectures, and document sources. We evaluate with a type-aware methodology that scores every field as a correct extraction, a miss, a hallucination, or a correct absence, so that declining to extract is distinguished from extracting wrongly. Our central finding is that aggregate accuracy conceals critical divergence along every axis: synthetic documents overestimate accuracy by 38–45pp on insurance certificates (n=18 real documents); three models score within 1pp overall but diverge by 31pp per category; and two pipeline architectures diverge by 26pp on contracts—with the advantage not predicted by document length. An independently built extraction system, run under a protocol fixed in advance, agrees with our ground truth on 82–87% of fields across three models spanning two providers, providing evidence that the corpus is not calibrated to a single implementation. The corpus and evaluation tooling are released to enable reproducible comparison.

LLMs increasingly act as purchasing agents, which makes the LLM, not the user, the one choosing among the options that satisfy a request; its preferences quietly fix what gets bought and what it costs. Hotel booking is a clean instance: a high-volume choice settled on a few comparable attributes, where the pick reveals those preferences. We introduce PriceBench, a diagnostic benchmark that recovers an LLM’s price, quality, and brand preferences from its booking choices with a logit choice model, applied to 28 LLMs from 8 providers on 3,600 hotel tasks from 179 real New York City properties. We find that capability is associated with how consistently an LLM chooses, not with what it chooses: more capable LLMs hold stronger, more consistent preferences, while weaker ones either lock onto one position, exploitable by whoever controls listing order, or choose almost indifferently. What those preferences favor varies sharply across providers and even within one family: price sensitivity spans more than an order of magnitude, and the price/quality trade-off moves mean booked nightly price from USD 247 to USD 393 on identical tasks. What an agent buys must therefore be measured per LLM, not inferred, and we release the tasks, code, and all 28 response sets.

The explosive growth of short videos has presented significant challenges to content governance. Traditional manual labeling is inefficient, slow to iterate, and costly. Therefore, we explore and develop a multi-stage autopilot labeling framework aimed at progressively increasing the level of automation in short video content annotation. To move beyond the heavy reliance on human expert labeling at the initial L0 stage, we first introduce a discriminative MLLM labeler at the L1 stage, which is fine-tuned with supervised data to directly predict labels for short videos. Advancing to the L2 stage, we shift to a domain-adaptive MLLM pretrained on short video data and use it for zero-shot labeling, enabling flexible generalization without task-specific supervision. In the L3 stage, we propose an agentic annotation workflow, and equip the agent with several key tools, which empower the agent to handle complex, unfamiliar, or vaguely defined cases, significantly reducing the need for human intervention. We conduct extensive experiments across various content issues, and results demonstrate that each stage of the autopilot labeling framework progressively improves labeling accuracy, and has significantly reduced manual labor costs in real-world applications by up to 85%. This automated labeling approach is poised to gradually replace manual annotation, emerging as the new paradigm for short video content labeling.

Practitioners across industries increasingly use large language models to extract information from unstructured documents such as financial filings, legal contracts, and sustainability reports, yet typically rely on generic online recommendations for prompt design that lack empirical grounding in their target setting. We address this gap with a practical methodology for systematically varying individual prompt components (prompt structure, contextual information, and model selection) and measuring their effect on extraction quality. We use greenhouse gas emissions extraction from corporate sustainability reports as a case study of fixed numeric extraction from high-stakes PDF documents that vary widely in structure and presentation and are dispersed across company websites. Instantiated in a retrieval-augmented generation pipeline, our evaluation shows that best practices from industry-facing prompting guides do not always hold: a structured prompt incorporating roles, definitions, extraction rules, and JSON formatting unexpectedly underperforms a simpler question-style prompt, while the benefit of adding tabular context proves both prompt- and model-specific. We condense our evaluation procedure and findings into a practitioner’s decision guide for systematically testing design choices on task-specific data rather than adopting generic recommendations by default.

Semantic caching cuts LLM inference costs by serving a cached response to semantically similar queries. Standard practice evaluates these systems using PR-AUC, a metric that only measures how well scores rank and ignores whether they are usable at a fixed threshold. We show this mismatch leads to systematically poor deployment choices, as models with the highest PR-AUC are often the worst in operation. We introduce Precision-Cache Hit Ratio (P-CHR) AUC, a cache-aware metric that measures precision across cache utilization levels, and Operational Retention Rate (ORR), which captures how much offline ranking quality survives at deployment. We decompose the operational gap between offline and deployed quality into a recoverable threshold-utility component and an irreducible structural component fixed by the dataset’s positive rate. Our experiments show that the threshold-utility gap is governed by the training objective rather than data scale, and yields only to re-normalizing scores over the candidate pool or changing the training objective. Ultimately, model selection for semantic caching is a threshold-utility problem, not a ranking one, and measuring it is the first step to closing the gap.

As LLM-based shopping agents enter production, existing benchmarks fail to capture how a shopper’s requirements arrive: stated implicitly in the query, recorded in a profile, or revealed only when the right question is asked. Benchmarks that expose full intent upfront and grade only the final choice can neither pose this long-horizon challenge nor explain which requirement an agent missed. To address this gap, we introduce EComAgentBench, a benchmark of 662 tasks grounded in real Amazon products and reviews. Each task scatters these requirements across a visible query, a tool-gated profile, and scripted clarification; an agent must uncover hidden intent, verify candidates against attributes and review evidence, and commit to a single product within 100 tool calls. Moreover, typed, source-tagged rubrics grade every task, attributing each failure to a requirement and its source. Construction is automated yet reliable, with every answer fixed in code before any text is generated and every sample validated. Our evaluation of seven models reveals that even the strongest attains only 57.1% overall accuracy, and rubric satisfaction degrades from visible to hidden sources. Overall, we believe EComAgentBench will serve as a reproducible foundation for moving shopping agents from single-query search toward dependable assistance over long horizons.

In our deployed travel-planning service, most users give minimal inputs or free-form requests rather than the structured constraint checklists assumed by existing benchmarks. We therefore present TripScore, a behavior-grounded benchmark and evaluation framework built from real user logs and calibrated against 1,468 pairwise judgments by 203 travel experts. TripScore couples a hierarchical feasibility gate (format and commonsense) with a unified, point-wise reward that aggregates soft quality and preference fulfillment. Using TripScore as both evaluator and reward signal, we benchmark direct prompting, test-time compute, neuro-symbolic solvers, code agents, and fine-tuning. We find that reinforcement learning fine-tuning (e.g., GRPO) provides consistent gains over other approaches under the same base model and practical latency.

Image-based AI assistants are now deployed at production scale on e-commerce platforms, where a single uploaded image can trigger fundamentally different user intents: product search, style recommendation, visual encyclopedia, or utility tool calls, each demanding its own response format, tool invocation, and domain knowledge. Without per-intent behavioral constraints, LLM-based systems conflate these heterogeneous modes and fall short of domain quality standards, while the breadth and dynamism of the intent space render manual engineering infeasible. To address this, we present SkillChain, which closes the production feedback loop on Skill evolution, automating the lifecycle of Skills through three stages: Skill Creator for bootstrapping from task specs and trajectories, Route Optimizer for routing alignment, and Body Refiner for iterative Skill Body refinement via dual-path LLM-Judge evaluation. Deployed on a production-scale e-commerce image assistant, SkillChain substantially improves aggregate response quality, with the strongest gains on structural compliance and content quality; a one-week online A/B experiment further confirms significant gains in user engagement, content consumption, and long-term retention.

Long-context and RAG workloads expose production bottlenecks that raw tokens/s and offline accuracy do not capture, including prefill cost, KV-cache pressure, prefix reuse, routing, and latency SLOs. We introduce OpenDC-Infer-LC, a hardware- and backend-neutral benchmark for evaluating complete data-center LLM serving stacks under long-context and retrieved-context workloads. The benchmark ranks systems by SLO-qualified, quality-preserving goodput: valid output-token throughput at the highest load satisfying p95 TTFT, p95 TPOT, p95 end-to-end latency, and success-rate targets. Because RAG and LongBench quality depends on model ability, retrieval difficulty, and task difficulty, OpenDC-Infer-LC uses a relative quality guardrail against a low-load reference run for the same model and workload, while reporting absolute quality separately. Pilot results on AMD MI355X and NVIDIA B200 show that deployable capacity depends strongly on backend scheduling, cache reuse, context length, routing, quantization, and runtime maturity.

Rubric generation improves the interpretability and actionability of large language model evaluation, but in online-to-offline (O2O) local-life AI search, user queries often reflect more complex needs, such as consumption intent, location constraints, and scenario preferences. Directly generating rubrics from a single query tends to produce generic criteria that fail to satisfy AI search evaluation needs, while leveraging candidate responses, preference samples, or external evidence at scale remains challenging. To address these limitations, we propose Category-Aware Rubric Generation (CaRG), a category-aware evaluation dimension candidate pool framework. Through embedding-based clustering, representative selection, and iterative LLM-based consolidation, CaRG constructs reusable candidate pools of both general and category-specific evaluation dimensions from representative historical queries. Subsequently, these candidate pools are leveraged to generate query-level rubrics with concrete positive and negative scoring criteria for target queries. Experimental results demonstrate that CaRG consistently outperforms Direct-Rubric and shows more stable overall quality than existing query-only baselines, while achieving the strongest overall agreement with production golden rubric rankings. Our repository is available at https://github.com/PlussLe/CaRG.

Al-powered people search platforms are increasingly deployed for recruiting, sales prospecting, and professional networking, yet no standardized benchmark exists for their rigorous evaluation. We present PEOPLESEARCHBENCH, an open-source benchmark comprising 119 multilingual queries across four scenarios: corporate recruiting, B2B sales prospecting, expert search, and influencer discovery. A central contribution is Criteria-Grounded Verification, an evaluation methodology that decomposes each query into explicit, independently checkable criteria and verifies each returned individual via live web search, producing factual relevance judgments rather than subjective LLM-as-judge scores. We evaluate four architecturally diverse platforms along three complementary dimensions-Relevance Precision, Effective Coverage, and Information Utility-and find that multi-source search agents significantly outperform single-domain systems, particularly in influencer discovery where the performance gap is largest. Platform rankings are robust across ablations on scoring thresholds, dimension weights, and judge models. All code, queries, and evaluation prompts are publicly available.

We introduce Telco-GAIA, a bilingual, multi-modal benchmark for evaluating tool-using agents on the data of a real-world telecommunications operator. Telco-GAIA comprises 100 human-verified question-answering tasks, in English and Arabic, that each demand multi-hop reasoning (4.2 hops on average) over three heterogeneous sources: a static website snapshot (HTML, images, and linked PDFs), a synthetic relational SQL database, and external web archives, spanning text, image, and tabular modalities. The benchmark is delivered as a sandboxed Docker environment and scored by normalized exact string matching, making evaluation objective, deterministic, and reproducible over time without any LLM-as-a-Judge. Evaluating a purpose-built reference agent across twelve commercial and open LLMs, we find Telco-GAIA challenging: even the strongest model solves only 71% of tasks; under a moderate cost budget, this falls to about 40%, and the visually grounded categories remain the weakest, where the average backend scores below 30%, leaving substantial headroom in document and image understanding. Telco-GAIA offers a rigorous, reproducible testbed for enterprise agents and a template for constructing closed-domain benchmarks.

Agentic pipelines assign a separately optimized system prompt to each step, so the cost of automatic prompt optimization (APO) multiplies across the pipeline. In this setting, the strategy for selecting candidate prompts cannot ignore budgetary constraints. We study the selection step of a textual gradient APO framework (Pryzant et al., 2023) in financial question answering with retrieval-augmented generation, a domain that stresses both the prompt and the reward signal. We compare the framework’s stateless Upper Confidence Bound (UCB) selector against two learned alternatives, PPO and DPO, with BEM as the shared reward. Across 135 experiments spanning three benchmarks, three budgets, and five random seeds, the selectors differ little in average accuracy gain (UCB 13.5% vs. DPO 13.4% vs. PPO 11.2%). Averaging over seeds erases these gaps and cost becomes the differentiating factor. PPO runs in under half the time of UCB (76 vs. 182 minutes per experiment) and its cost stays nearly flat as budget grows. For cost-sensitive deployments, selection has to consider efficiency in addition to accuracy, and PPO emerges as a promising low-cost selector within this APO setup. This design was shaped by an in-progress deployment at a consulting firm that assigns one system prompt to each step of an agentic data-cleaning pipeline.

Enterprise AI assistants must produce responses that are verifiable and traceable to source evidence. However, retrieval-augmented generation (RAG) over heterogeneous enterprise data can suffer from citation drift, unsupported content, and weak source traceability. We present EvidenT (T = Trust + Transparency + Traceability), a lightweight pipeline that verifies extracted evidence against retrieved documents before answer generation, without model retraining. EvidenT combines structured passage extraction with deterministic lexical alignment to filter unsupported content, correct citation drift, and preserve source-span traceability. On approximately 500 real enterprise queries, EvidenT improves gold-source hit rate by an average of 29% over prompting baselines, produces no citations to non-retrieved urls, and achieves near-saturated answer-to-source lexical coverage.

Large Language Models (LLMs) remain vulnerable to adversarial prompts that exploit safety mechanisms. Existing red-teaming methods face scalability challenges, computational bottlenecks, or limited attack diversity. We propose , a framework that reconceptualizes adversarial prompt generation as evolutionary quality-diversity search, where diverse attack strategies co-evolve across behavioral niches. introduces two synergistic innovations: (1) multi-element archives that maintain populations of elite solutions per niche, and (2) parallel fitness evaluation that replaces pairwise comparisons with efficient probabilistic scoring, achieving 𝛩(M) speedup (from 𝛩(M²N) to 𝛩(MN)). Experiments demonstrate superior performance: compared to Rainbow Teaming, generates 100× more prompts (10,418 vs. 100) with higher attack success rates. Against nine state-of-the-art methods on HarmBench with 12 LLMs, achieves 81.1% average ASR - surpassing AutoDAN-Turbo by 3.9 points - while being 9× faster (1.45 vs. 13.50 hours). Code: https://github.com/knoveleng/rainbowplus

Matching candidates to vacancies is central to recruitment, and a recruiter needs to see why a candidate fits, not only a single opaque relevance score. We provide this evidence as named, interpretable matching dimensions recruiters can act on — eight in our current deployment. We propose a two-part approach. The first is an LLM-based labeler whose prompts and feature definitions were refined from recruiter feedback while it served as an earlier production matching stage. In the current architecture, it is used only for offline labeling and is not called on online requests. The second is a feature bi-encoder distilled from it: a LoRAadapted embedding backbone with compact per-dimension heads that runs on CPU and serves all online requests. Both parts keep improving: prompts are revised as feedback arrives, and the bi-encoder is retrained on the updated labels. The model is trained on 168,772 labeled vacancy–resume pairs (17,921 vacancies and 180,030 resumes). Recruiters using the service can confirm or revise surfaced feature predictions. On 927 recruiter-recorded values from this selected production-feedback subset, the deployed student agrees with the recorded decisions in 888 cases (95.79%). This is operational, non-blinded agreement rather than an independent human evaluation.

Loan origination is the process by which a lender creates a new loan, from application and underwriting through approval and funding. This process serves a critical role in evaluating the eligibility and level of risk posed by an applicant. Recently, firms have begun using mortgage loan agents to augment human loan officers, despite a lack of any public benchmark. To fill this gap, we present MortarBench, a loan origination agent benchmark. MortarBench uses a financial data synthesis and mutation pipeline to generate examples with broad edge case coverage that match real-world distributions and questions. We find that state-of-the-art large language models (LLMs) perform poorly, with closed-source models achieving at most 80.8% F1 accuracy. We also discover systematic biases in LLM perception of foreignness related to non-English names. Noting these weaknesses, we introduce CRIT, a confidence calibration framework. Our method increases peak F1 accuracy to 83.6% while improving risk management steering and reducing bias.

Dense retrieval is a fundamental component of e-commerce search, projecting user queries and items into a shared semantic representation space for large-scale, real-time retrieval. While LLM-based embedding models are increasingly replacing BERT-style encoders, prevalent training pipelines still largely rely on BERT-like supervised fine-tuning and hard negative mining. Such paradigms require intricate offline negative construction, thereby constraining model iteration efficiency and limiting the evolution of learned semantic representations. We propose Retrieval-GRPO, a dense retrieval framework based on reinforcement learning. It dispenses with offline hard negative mining by performing dynamic Top-K candidates retrieval during training. Moreover, we introduce a relevance-oriented LLM as a reward model to provide real-time supervisory signals. Guided by the resulting reinforcement learning objective, the retriever updates embeddings to better capture semantic correspondence and achieve improved generalization on complex long-tail queries. Extensive offline and online experiments substantiate the effectiveness of Retrieval-GRPO in Taobao Search, which serves China’s largest e-commerce platform.

Commercial design platforms increasingly edit documents through large language model (LLM) agents, but two practical problems block reliable deployment: legacy document formats expose only flat, absolutely positioned elements, so agents must recompute coordinates and routinely break layouts; and design has no unique ground truth, so diff-against-reference metrics penalize valid-but-different outputs. We present ACE, an agentic canvas editor over a hierarchical scene-graph with a presentation-specialized action space (98 tools), paired with CARE, a content-aware router that feeds the agent only the relevant slice of each deck (avg. ∼ 89% input-token reduction), and a self-correction loop driven by a ground-truth-free instruction-following (IF) judge whose natural-language critique is fed back as the next-turn instruction. On a multi-slide benchmark and with a fixed backbone, a scene-graph editor in a single turn already matches a same-backbone agentic HTML pipeline that iterates internally; adding self-correction lifts ACE clearly above it on quality (IF 4.45 vs. 4.09; VQ 4.02 vs. 3.89) while remaining 1.75× faster and ∼ 44% cheaper. Self-correction contributes +0.41 IF / +0.27 VQ, and 66% of cases halt after one pass, keeping it cheap.

Production Retrieval-Augmented Generation (RAG) systems serve repeated queries over evolving document corpora under user-specific access constraints. Standard pipelines perform full retrieval and generation for every request, wasting computation on recurring queries. Naive caching reduces latency but introduces safety failures: stale evidence from updated documents, unauthorized content crossing access boundaries, and superseded passages no longer reflecting source truth. These violations are invisible on academic benchmarks yet unacceptable in enterprise deployment. We introduce SSC (Safe and Sufficient Caching), a validated evidence reuse framework for production RAG. SSC caches evidence chunks (not generated answers) with perchunk provenance metadata, and enforces hard validity constraints before reuse: access-control gating ensures user authorization, and freshness verification confirms source-document currency. Only evidence passing both checks enters a utility-based ranking stage balancing relevance, source trust, historical utility, and serving cost. This separation of safety from efficiency is the central contribution. On WebQuestions and HotpotQA, SSC maintains full quality parity with unrestricted retrieval baselines, confirming zero degradation from evidence reuse. On a proprietary InternalQA workload with role based access control and document versioning, it reduces end-to-end latency by 22% and retrieval cost by 34% while achieving zero invalid reuse, compared to 3.4-9.8% violation rates for standard caching and 0.3-8.2% for TTL-based invalidation. The method is training-free, retriever-agnostic, and deploys as a thin validation layer in existing RAG pipelines.

Organizing unstructured feedback text into hierarchical taxonomy is a fundamental challenge in NLP, particularly in domains where feedback arrives at massive scale in varied forms such as reviews, transcripts, and surveys. Existing approaches either produce shallow hierarchies, neglect long-tail topics, or lack rigorous evaluation frameworks. We present TaxCE, a fully automated framework that constructs multi-level hierarchical taxonomies from raw text through progressive condensation of corpus content into actionable segments, deduplicated semantic units, and granular topics with definitions, which are then organized bottom-up into a hierarchy with corpus-groundedness. We also introduce three corpus-grounded evaluation metrics, Exclusivity, Exhaustivity, and Granularity (EEG), and integrate them into a metrics-in-the-loop iterative refinement mechanism that diagnoses deficiencies and applies targeted corrections until convergence. Extensive experiments demonstrate that TaxCE consistently outperforms existing baselines spanning classical topic models, neural methods, and LLM-based approaches, with average improvements of 11.8, 20.5, and 15.7 percentage points in exclusivity, exhaustivity, and granularity respectively over the strongest baseline. Human evaluation further confirms superior taxonomy quality, actionability, and navigability.

Effective manager-employee communication is critical for retaining high performers and developing underperformers, yet training managers in these skills remains costly. Text-based chatbots offer a scalable approach but cannot provide realistic rehearsal: managers need to practice speaking aloud to build confidence before high-stakes conversations. In this paper, we propose Conversation Coach, a voice-first AI system that enables managers to rehearse difficult workplace conversations in a realistic spoken format. The system addresses three challenges: achieving low-latency interactions with strong language understanding, enabling adaptive conversations through configurable bot personalities that simulate different employee types, and generating personalized feedback on content and policy compliance. We compare an end-to-end speech-to-speech model with a cascaded approach combining automatic speech recognition, a large language model, and text-to-speech synthesis. The end-to-end approach achieves 3× lower median (P50) latency with native barge-in capability at an estimated 8× lower cost, while the cascaded approach offers superior reasoning essential for coaching quality. We deployed the cascaded architecture in production, where 40,000+ managers used it over six months, with adoption patterns indicating selective use for difficult conversations.

Production tool calling for agentic systems must satisfy four operational requirements: bounded per-invocation cost, predictable latency, adaptability to evolving tool catalogs, and the ability to catch errors before execution. The ReAct paradigm keeps the large language model (LLM) in the loop, but at production scale, re-processing intermediate results inflates cost and latency and degrades answer quality. Programmatic tool calling (PTC) offers an alternative; the LLM generates a single script that resolves dependencies and extracts relevant fields in one inference pass. PTC is a harder generation task, however, because the LLM must produce code that handles complex, variable responses without observing intermediate results, and existing remedies each fail at least one of the four requirements. We describe a production deployment of PTC serving 26,528 invocations, paired with a three-layer abstract syntax tree (AST)-based validator that rejects invalid scripts before execution and requires no training data, preserving adaptability to evolving tool catalogs. This reduced average input tokens by 77%, latency by 21%, and increased the answered-query rate by 5.6 percentage points across 19 production agents.

A/B testing is the gold standard for evaluating product changes, but each experiment requires real user traffic, engineering effort, and weeks of measurement. We propose a simulation framework that predicts A/B test outcomes using LLM-powered agents conditioned on data-driven personas grounded in real user behavioral signals. Unlike prior work that relies on synthetic or rule-based personas, our agents are constructed from anonymized behavioral data—activity patterns, engagement signals, and inferred demographics—enabling more faithful population modeling. We frame A/B test simulation as a structured question task and systematically study (i) question design formats, (ii) the impact of persona data source and domain alignment, (iii) the trade-off between per-persona behavioral depth and population diversity, and (iv) efficient population subsampling. On a benchmark of 40 A/B tests spanning two metric types, our best configuration achieves 0.75–0.90 directional accuracy depending on the test metric, demonstrating that data-driven personas are a viable path toward fast, low-cost experiment pre-screening.

The quality of video subtitle translation cannot be assessed from sentence-level semantic equivalence alone. Subtitles are governed by audiovisual cues, speaker relationships, and narrative context that accumulates across scenes, yet existing automatic metrics and text-centric LLM-based evaluation do not reflect these factors.

We propose SCR-Eval, a context-aware evaluation framework for video subtitle translation, built on a Script-like Context Representation (SCR) reconstructed from the video. The evaluator aligns the translated subtitles to the SCR’s semantic scenes, then performs MQM error analysis with an LLM judge.

On three open movies, we benchmark five English-to-Korean subtitle providers against blind, randomized annotations by seven professional evaluators. The framework’s provider ranking agrees strongly with the experts (system-level Spearman ρ = 0.967). In ablations, agreement weakens stepwise from full context through scene-bounded text to isolated source–translation pairs. Automatic baselines agree less: MetricX and xCOMET correlate negatively with the expert ranking, while RUBRIC-MQM correlates only weakly.

With full context, SCR-Eval shows substantially stronger agreement with experts than these baselines.

Personalized LLM assistants must recover sparse evidence from long conversation histories across queries of varying complexity. We introduce APDMem (Agent-controlled Progressive Disclosure Memory), a hierarchical long-term memory architecture that applies progressive disclosure to memory retrieval. Rather than relying on a flat memory store or fixed retrieval granularity, APDMem represents conversation history as four progressively detailed layers: thematic summaries, personalized key facts, turn-level evidence notes, and raw messages. At inference time, a controller applies progressive disclosure to the memory hierarchy: it first reads high-level summaries and drills into finer evidence only when needed. This creates an adaptive cost-fidelity trade-off: simple queries can terminate early, while complex temporal, multi-hop, or exact-evidence queries trigger deeper inspection. A note synthesizer converts retrieved evidence into a query-focused structure that consolidates facts, orders events, and flags contradictions before final answer generation. Experiments on LongMemEval show that APDMem achieves strong performance for long-context memory reasoning while accessing only 8% of the total conversations.

Production LLM deployments maintain heterogeneous model pools spanning order-of-magnitude cost differences, yet existing routers make binary strong-vs-weak decisions coupled to specific model identities, requiring retraining on every catalog change. We present HyDRA (Hybrid Dynamic Routing Architecture), which uses a lightweight ModernBERT encoder to predict capability requirements (reasoning, code generation, debugging, and tool use) for each query. It then selects the cheapest model meeting those requirements through shortfall matching. HyDRA decouples the learned predictor from the model catalog, enabling catalog changes without retraining. On SWE-Bench Verified, HyDRA is 0.3 QR points below always-strong (not statistically significant) at 54.1% cost savings. On a held-out multilingual evaluation set, the same checkpoint retains 80.6% of oracle-routing quality at 37.5% cost savings, tunable to 79.1% retention at 56.1% savings. A 14-day A/B flight with close to 1M users per arm found statistically significant improvements in latency, time-to-first-token, reliability, and user activity. Deployed across GitHub Copilot VS Code Chat, GitHub Copilot CLI, and GitHub Copilot extensions, HyDRA demonstrates cross-lingual routing consistency across 16 languages and 4 language groups.

Pharmacokinetic (PK) parameters are critical to drug development, but many remain embedded in free-text biomedical literature rather than a structured form. Existing extraction approaches primarily target explicit parameter mentions, while a substantial proportion of PK values must be inferred from context. We introduce implicit PK parameter extraction, where parameter identity is not explicitly stated but must be resolved through domain-grounded, multi-sentence reasoning. To address this challenge, we propose CORVA, a validation-enhanced LLM-based extraction framework that integrates reasoning and domain constraints to extract structured PK tuples from text. We also construct a labelled dataset from DrugBank absorption descriptions, of which 47.5% involve implicit expressions. Relative to the strongest baseline, the proposed method improves F1 by 35.3% and tuple accuracy by 52.9%. CORVA achieves the highest implicit recognition rate at 45.8%, which is 10.5 points ahead of the runner-up. Our results demonstrate that reliable implicit PK extraction requires validation as a first-class component rather than a post-processing step, which offers a practical framework for scalable knowledge extraction from biomedical literature. The code and data are available at https://github.com/wuqinhao/CORVA.

Evaluating LLMs across many model variants—quantized, fine-tuned, or deployment-specific—requires running large benchmarks repeatedly, a process that can take tens of hours per model on edge hardware such as NPUs. Existing subset selection methods reduce this cost but depend on large calibration pools or learned prediction layers. We introduce MINCE (Monte Carlo Informed N-sizing for Compact Evaluation), which uses Monte Carlo simulation over per-item logs from a small set of calibration models to find the minimum subset size that bounds accuracy drift and then fixes a randomly sampled subset at that size, with no prediction layer needed. MINCE reduces IFEVAL by 54%, MMLU by 89%, GSM8K by 70%, and MMLU-Pro by 88% with maximum drift ≤2.62 pp on BF16 calibration models. The frozen subsets generalize to held-out GPU models with mean drift ≤1.40 pp and to INT4 NPU models with mean drift of 0.77–3.59 pp, while delivering evaluation speedups of up to 8.1× on the GPU models and evaluation speedups of 1.7–3.4× on the NPU models. The method is robust to calibration pool size and achieves lower drift than tinyBenchmarks (12× lower on MMLU, 3.3× on GSM8K) while using 42× fewer calibration models.

Million-token context serving is becoming a critical requirement for industry LLM systems for real deployments. However, practical 1M-context serving for open-source models has remained heavily CUDA-centric, largely because the required long-context runtime features are not fully available on AMD ROCm. With AMD GPUs now reported at around 19% market share, this gap matters. We address it by enabling 1M-token Qwen3 serving on AMD GPUs with ROCm. We present , a runtime stack that enables million-token serving for the widely used open-source Qwen3 family on AMD ROCm. achieves this by intercepting vLLM’s attention routing at runtime and redirecting queries, keys, and values to JIT-compiled Triton Dual Chunk Attention (DCA) kernels, without forking the serving engine. We then harden the surrounding runtime to make full 1M-token execution practical: DCA-aware paged metadata, tiled single-request chunked prefill, MoE token chunking that reduces per-rank transient memory from approximately 91.6 GiB to 10.6 GiB, and FP8 GEMMs with hardened hipBLASLt kernel selection. We evaluate across multiple million-token context tasks and show consistent serving performance and exact retrieval correctness. To our knowledge, this is the first reported end-to-end Qwen3 million-token serving path on AMD ROCm integrated with vLLM.

E-commerce search systems typically separate query-item relevance matching from downstream personalized ranking. While effective for large-scale serving, this design leaves relevance levels dominated by objective query-item matching, forcing follow-up ranking to operate over candidate structures that may not reflect user behavioral preference. We define a new personalized behavioral relevance task, which predicts whether a relevance-stage candidate item is supported by the user’s behavioral evidence under current query intent. We develop a corresponding framework, BRIDGE, for Behavioral Relevance Inference with Dynamic Guidance from Evidence. It organizes multi-timescale behavioral context from online logs, obtains reliable supervision with multi-perspective annotation and adjudication, and trains an LLM to produce structured evidence-grounded decision rationales. It particularly incorporates sample-specific rubric rewards into GRPO to improve generated reasoning traces. Offline experiments validate that BRIDGE improves both prediction accuracy and reasoning quality. Online A/B tests show gains of 0.19% in active user-days and 0.67% in transaction-related user-days, both significant in statistics.

Automotive-finance screening can fail when individually valid documents are paired with an internally inconsistent applicant narrative. We present the Adversarial Cognitive Probing Framework (ACPF), a theory-motivated staged protocol that scores four ordinal behavioral dimensions and delegates the final decision to a deterministic backend predicate. We evaluate ACPF on a risk-enriched retrospective corpus of 340 officer–applicant dialogues: 72 institutional adjudications of suspected misrepresentation, 138 cases of significant early delinquency, 110 good cases, and 20 non-fraud rejections excluded from the primary metrics. On identical fixed transcripts and the same QWEN3.5-122B carrier, structured scoring raises combined-risk capture from 5.7% to 61.0% while reducing the false-rejection rate from 14.5% to 12.7%. With OmniMAX, the selected low-FPR carrier, ACPF captures 58/138 (42.0%) delinquency cases at 6/110 (5.5%) false rejections; current origination review flags none of these delinquency cases. On the institutionally adjudicated stratum, ACPF captures 56/72 (77.8%) versus 72/72 for current review, but we make no comparative claim because the label and reference decision share institutional provenance. A shadow deployment has no adverse-action authority; prospective outcome validation is not yet available.

Fine-tuning of Large Language Models (LLMs) using Low-Rank Adaptation (LoRA) on an end-user’s data offers personalized experiences while keeping data private, but faces severe memory constraints on consumer hardware. Peak memory during fine-tuning often exceeds device limits, especially for models with billions of parameters and long-context training data. This paper introduces a suite of complementary techniques to reduce memory footprint: (1) base model quantization with on-the-fly dequantization, (2) memory-efficient checkpointing combining selective activation caching and disk offloading, (3) softmax approximation using semantically relevant token subsets, and (4) logits masking. Only softmax approximation alters the training objective and we discuss the resulting quality trade-off both analytically and empirically. Experiments on Llama-3.2 3B and Qwen-2.5 3B demonstrate up to 26× and 28× reduction in peak memory, enabling fine-tuning on resource-constrained devices.

We argue that robust alignment for non-verifiable tasks is fundamentally a principle generalization problem: reward should be guided by explicit, auditable principles rather than compressed into a learned scalar function. To operationalize this view, we present the Open Rubric System (OpenRS) , a modular, rubrics-based LLM-as-a-Judge framework built around Pairwise Adaptive Meta-Rubrics (PAMR) and lightweight Pointwise Verifiable Rubrics (PVRs). OpenRS is governed by an explicit meta-rubric—not a rubric itself, but a higher-level guideline that specifies how rubrics should be generated, which criteria to prioritize, and which failure modes are non-negotiable—and instantiates concrete adaptive rubrics on the fly by conditioning on the semantic differences between two candidate responses. It then performs criterion-wise pairwise comparisons and aggregates criterion-level preferences externally, avoiding pointwise weighted scalarization while improving discriminability in open-ended settings. Finally, we instantiate OpenRS as reward supervision in pairwise RL training. Empirically, OpenRS achieves state-of-the-art results on four reward-modeling benchmarks and yields stable gains when replacing scalar reward models in end-to-end RL.

Document RAG retrieves from parser outputs, yet parsers are commonly selected by intrinsic measures of transcription fidelity and boundary quality rather than by retrieval performance. We introduce the Retrieval-Conditional Parsing Score (RCPS), a training-free protocol that ranks parser–chunker combinations on a fixed held-out Q–A retrieval probe. RCPS averages mean reciprocal rank (MRR) across specified retrievers and retrieval depths, using the same format-normalised exact-span relevance rule for every candidate. On KoGovDoc-RAG, table-enabled MinerU-on has a higher Boundary Clarity score than our production parser, Prod (0.713 versus 0.610). Yet its Hit@1 is 42.6 percentage points lower (0.123 versus 0.549). Across five parsers evaluated on all 294 pages, RCPS ranges from 0.137 to 0.584. With Prod fixed, the four chunkers score between 0.535 and 0.593. A retriever-free coverage diagnostic finds no normalised exact match for 20.2% of reference spans in Prod output. No tested chunker splits more than 2.3%. On source-aligned OHR-Bench data, semantic noise lowers retrieval while Boundary Clarity remains stable or changes non-monotonically. In a secondary parser-training study, two direct preference optimization checkpoints improve Hit@5 by only 0.95 and 1.15 percentage points over Prod on an audited compatibility subset; a matched edit-distance control performs similarly. The workflow therefore selects with RCPS, diagnoses with coverage, and trains only if needed. We release the 663-Q–A KoGovDoc-RAG probe and an RCPS reference implementation.

In e-commerce, product detail pages (PDPs) often contain hundreds or thousands of customer reviews, yet the interface can surface only a small set of concise and reliable insights. A useful review aggregation system must therefore abstract recurring themes while preserving which reviews support each insight. We study this problem as structured review-insight mining: review-level tuples are first extracted and then aggregated into ranked product-level insight clusters with explicit member traceability. We propose GraphReduce, a coverage-preserving framework for product-scale review insight aggregation. It combines a contrastively trained projection layer over frozen sentence embeddings, graph-based clustering on local neighborhood structure, and an LLM reducer/ranker for naming and ordering canonical clusters. By keeping membership accounting outside the generative step, GraphReduce preserves complete tuple coverage and avoids a common failure mode of direct LLM summarization: producing plausible high-level themes while silently dropping long-tail evidence. Across three product-category benchmarks, GraphReduce improves recovery of gold insight clusters over strong lossless embedding-based baselines while maintaining comparable ranking quality. We further show that flat LLM and topic-modeling baselines can appear competitive under ranking metrics despite omitting substantial input evidence, and that an anti-copy reinforcement objective improves the quality of upstream tuple extraction. These results highlight the importance of coverage-aware aggregation for faithful review insight mining in e-commerce settings.

Generative query suggestion aims to enhance user engagement by anticipating user intents and recommending relevant follow-up queries. A central challenge is to generate slates whose individual queries are useful while the slate covers distinct intents. We propose an Intent-Driven Query Suggestion Framework with dual-stage optimization. First, intent-aware diversity modeling constructs intent-aligned supervised fine-tuning (SFT) data and uses an Intent-Aware Diversity Reward to optimize intent coverage. Second, query-level credit assignment routes individual quality signals to the corresponding query tokens while sharing a slate-level diversity signal across the slate. Experiments on a large-scale production dataset, including online A/B testing and offline evaluation, show improvements in click-through rate, query quality, and intent coverage.

Extracting structured information from visually rich documents at enterprise scale demands both the reasoning capability of large language models and the efficiency of deterministic execution. Current approaches either deploy LLMs as instance-level analysts, incurring per-document inference costs that are prohibitive for repetitive templates, or rely on manually authored vendor-specific prompts that do not scale beyond a handful of high-volume vendors. We present CACHE-ED2, a framework that repositions the LLM as a system-level developer: given a novel document format, a multimodal ReAct-based agent synthesizes a reusable Document Extraction DSL (DocExDSL) program encoding the complete extraction logic (strategies, disambiguation, transforms, and automated reasoning validation), then caches it for deterministic, LLM-free execution on all subsequent documents of the same format. A contrastive Document Format Encoder routes incoming documents to cached programs with perfect hit rates, and a HITL Feedback Analysis Agent incorporates analyst corrections to refine programs over time. On public benchmarks (CORD v2, FATURA) and production streams spanning seven business verticals (580,000+ documents, 11,500+ vendors), CACHE-ED2 achieves near-perfect extraction accuracy (∼0.99) with a 1.0 extraction rate on document-present fields and perfect format matching across 50+ benchmark templates, matching or exceeding manually authored vendor-specific prompts, while reducing cumulative token consumption by ∼2.6× over first 1,000 documents.

We present AgentLens, a trajectory-level evaluation framework for interactive code agents, built from production-informed tasks and integrated into continuous regression testing. Most code-agent benchmarks reduce a run to a single bit—did the task pass?—but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way. AgentLens evaluates that whole trajectory, pairing formal verification with LLM-written trajectory reviews and side-by-side comparisons, so that each run yields a readable explanation of the final score. Apart from ranking models, AgentLens can be used to diagnose model behavior, compare successive versions of agents, and catch product regressions in an evaluation pipeline. We release AgentLens at https://github.com/agent-lens/agent-lens-bench.

Aspect-based scientific summarization produces a focused summary of a long document for a given aspect (e.g., Approach, Methods). Prior work decomposes the task, assuming a model can neither attend over the full document nor resolve aspect complexity in one pass. We revisit both assumptions now that compact yet capable language models handle long contexts natively. Across three datasets and two primary open long-context models, decomposition provides no reliable quality advantage over direct full-document prompting. With the main 9B model, zero-shot is strongest on aggregate lexical, semantic, and claim-recall/F1 metrics; under the 7B robustness model, simple decomposed variants occasionally edge zero-shot, but gains are small and inconsistent. On these primary models, the multi-role agentic pipeline is never best in aggregate and incurs substantially higher token cost, whereas RAG uses roughly one seventh of zero-shot’s tokens but delivers lower, less consistent quality. We characterize where added complexity loses, quantify the cost–quality trade-off, and distill practical deployment guidance. Code is available on GitHub.

Generative search introduces failures in query matching, competitive ranking, and grounded generation, while conventional SEO and final mention metrics often cannot locate these failures. We present a controlled Stage-Isolated Evaluation framework that converts stage-specific errors into fact-grounded sitemap revisions. In an industrial study with an anonymized consumer-health company, each version is evaluated against nine fixed competitors using disjoint training, validation, and held-out test queries. The selected checkpoint improves the test score from 58.78 to 69.98, exceeding one-shot rewriting by 6.19 points and same-budget recursion without diagnostics by 3.07 points. Different stages favor different checkpoints, revealing non-monotonic trade-offs. Pilot online results and human annotation provide additional support for the framework as a reproducible diagnostic proxy for enterprise GEO.

A deployable multilingual reranker must not only generalize across languages, domains, and ranking tasks, but also remain efficient to serve as a second-stage reranker in practical systems. However, adapting it to new target distributions typically requires extensive task-specific relevance annotations. We present Querit-Reranker, a family of multilingual rerankers trained with a data-centric pipeline for label-efficient adaptation. We instantiate it as Querit-Reranker-A0.4B, initialized from an in-house MoE backbone with 0.4B activated parameters, and Querit-Reranker-4B, initialized from Qwen3-Embedding-4B. Our pipeline first learns general relevance modeling from large-scale ranking-oriented data, then adapts to target distributions through synthetic-query mining with teacher scores as continuous soft labels. To consolidate complementary task-adapted strengths, we further merge checkpoints via spherical linear interpolation, obtaining a single deployable model without runtime ensembling overhead. Using Qwen3-Embedding-0.6B as the first-stage retriever, Querit-Reranker-A0.4B improves average nDCG@10 from 54.11 to 59.28 on BEIR and from 59.87 to 67.70 on MIRACL. On MTEB Multilingual v2 Reranking, it also shows competitive performance with larger embedding-based models, while Querit-Reranker-4B further achieves state-of-the-art performance among all evaluated models. We release both models on Hugging Face.

Cementitious materials teams need machine-readable databases of compressive strength records to support mix-design optimization. However, manual curation of the literature is too slow and hard to scale. We present CemLit-IE, a five-stage zero-shot pipeline, converting raw PDFs into fully attributed records with provenance. The core challenge is that the relevant information is distributed across tables, text, and figures. Even though table-only extraction produces plausible numbers, the results are not auditable. CemLit-IE addresses this through a series of methods, including keyword-weighted page scoring, structured table injection, value-provenance schema, and review queue. On a benchmark of 20 papers with 655 expert-annotated records, CemLit-IE achieves a recall of 76.3% with 99.2% precision and 100% exact-match accuracy in a text-only model. The recall can be raised to 83.5% when figure reading is enabled. The ablation study shows that each design choice contributes to improving precision. As for the cost, a separate 270-paper journal-batch run extracted 2,039 strength records at $31.28 total API cost. All data and code are publicly released.

Deploying LLMs for financial information retrieval demands accuracy, cost efficiency, and reproducibility, yet most approaches require expensive fine-tuning. PRISM is a training-free framework that composes three deployable modules (prompt engineering, in-context learning, and a candidate refinement module) with a boundary-condition study of multi-agent coordination for financial document retrieval. On FiQA-2018, stacked reranking and k-reciprocal encoding raise Recall@100 from 0.7783 to 0.8326. On FinAgentBench, a Cohere-refined top-50 pool yields a higher observed end-to-end NDCG@5, below this evaluation’s minimum detectable effect, while cutting per-query cost by about 35%. Added complexity does not always pay off: k-reciprocal encoding stops helping once the upstream stage is already high-recall, and multi-agent coordination is not a default for fine-grained chunk ranking. Matched pairs that hold prompt, model, and exemplars fixed lose 0.0702 and 0.0681 combined NDCG@5 to their single-path controls, and the whole gap sits at the chunk stage. We release an expert-annotated FinAgentBench evaluation set, together with a label-robustness sweep over 16 alternative ground-truth constructions, restoring a reproducible evaluation path after the original evaluator went offline. Without fine-tuning, PRISM placed third on the official FinAgentBench private leaderboard (leaderboard basis, NDCG@5 0.71181), reaches NDCG@10 0.6197 on FiQA-2018, and 98% accuracy on FinanceBench. We report per-query latency, token, and dollar accounting across roughly 40 configurations, present the quality–cost–latency Pareto frontier, and introduce FACET, a transparent SLA-specific scalarization for choosing among non-dominated configurations. Code and dataset are released at https://bit.ly/prism-ailens.

Online video content moderation is critical for protecting minors. Violations are increasingly implicit and context-dependent: a swimwear is innocuous in a vlog but high-risk with suggestive posing. Yet, traditional binary systems fundamentally fail to capture this continuous severity spectrum, directly leading to both excessive false positives for context-heavy samples and missing deeply implicit violations. We introduce KUROKO, a content moderation system that acts as a behind-the-scenes assistant, driving a paradigm shift in content moderation from binary classification to severity-aware ranking. Our contributions are threefold: (1) Human-Centric Severity Ranking: KUROKO shifts from rigid binary divides to a continuous risk spectrum, producing nuanced, expert-level rationales closely aligning with human judgment. (2) Verifiable Alignment and Interpretability: By integrating a severity-aware data engine and a verifiable reinforcement learning approach, we replace black-box decisions with transparent, actionable diagnostics. (3) Cost-Efficient Deployment: We engineer a coarse-to-fine inference architecture equipped with a lightweight recall classifier, enabling compact models to rival proprietary VLMs at a fraction of the computational cost. Extensive offline and online A/B tests demonstrate KUROKO’s advantages over traditional binary baselines and its competitiveness with leading commercial APIs at a fraction of their scale. Online, KUROKO yields more human-confirmed enforcement actions under an equal human-review budget, while lowering violative exposure and user negative feedback.

Industrial-grade distributed training of vision-language models (VLMs) remains far less efficient than that of unimodal LLMs. Existing solutions either follow a monolithic design that assigns uniform parallelism to heterogeneous modules or adopt a disaggregated deployment that separates modules while executing them as a batch-synchronized pipeline. In this paper, we highlight that the above solutions are still not sufficient, and VLM training can be further decoupled. To this end, we present FlowTrain, a flow-based decoupled training framework that reformulates VLM training as a producer-consumer dataflow coordinated through a unified memory pool. The encoder and backbone can progress independently over a global virtual address space. Since this execution decoupling fundamentally changes the optimization objective of allocation and scheduling, FlowTrain further introduces a heterogeneous parallel allocator that assigns module-specific parallelism strategies by solving a throughput matching problem. The dynamic packing scheduler is used to construct balanced microbatches at runtime according to the actual LLM-side computation cost. Extensive experiments on real-world workloads show that FlowTrain achieves over 50% MFU and up to 1.7× throughput improvement, narrowing the efficiency gap to LLM-only training.

LLM agents for structured data analysis rely on dynamic query generation, in which the model synthesises tool calls at runtime. This introduces failures such as fabricated dataset references, unsupported operations, and incorrect results, which existing mitigations reduce but do not eliminate. We propose predefined-tool execution as an architectural alternative in which the language model selects and parameterises tools from a fixed, schema-validated registry. We formalise the approach using typed tool specifications and prove executability and hallucination-suppression properties under assumptions. We complement the architecture with a Numerical Faithfulness Checker that verifies quantitative claims against pre-computed ground truth, and evaluate the system on a domain-specific benchmark using four language-model backbones. The suppression property follows from the validation layer under those assumptions, and held across all 276 evaluated agent executions involving 23 queries, four backbones and three runs. Mean exact-plan-match rates ranged from 50.7 to 53.6 per cent, with a between-backbone spread within a single backbone. Because exact plan match penalises valid but non-minimal plans, it is a strict lower bound on task correctness. Given only three runs per backbone, comparisons are descriptive, not evidence of equivalence The on-premise open-weight backbones performed comparably to commercial ones on the measured metrics, supporting confidential deployments.

Large Language Models (LLMs) are increasingly applied to telecom engineering tasks, yet perform poorly on 3GPP specifications. These standards encode much of their technical information in complex tables, but LLM knowledge and interpretation of such tables remain largely unexplored. We introduce TeleTables, a benchmark comprising 2,220 tables from 13 3GPP specifications in four formats and 500 human-verified MCQs spanning direct retrieval to multi-step reasoning. Evaluating 20 open-weight LLMs across non-reasoning, multimodal, reasoning, and table-specialized architectures reveals two distinct performance bottlenecks. In the closed-book setting, domain knowledge is the primary constraint, with no general-purpose model exceeding 41% accuracy. When the table is provided as context, the best models exceed 90%, but performance degrades systematically with reasoning depth, evidence scope, and structural complexity, with a 32.2 pp spread across reasoning skills. Table specialization on non-telecom data provides no consistent benefit, while strong reasoning capabilities remain essential for reliable interpretation of complex technical tables.

Financial-filings workflows depend on more than extracting reported values: each number must be placed in the document context that makes it reviewable, comparable, and auditable. In practice, this logical structure—including the relevant section, note, heading, row, column, period, currency, and surrounding text—is often clear to professional reviewers but only partially encoded in inline XBRL markup. As a result, rule-based parsers can work on familiar layouts yet fail silently when document structure departs from expected templates. We present FINSTRUCT, an on-premise LLM system designed for confidential financial-document workflows. FINSTRUCT runs locally on a single 16 GB GPU, requires no task-specific fine-tuning, predicts nine structural fields for each reported fact, and provides reviewer-facing rationales together with repeated-sample agreement signals for human-in-the-loop review. On 1,493 manually annotated facts from five UK FRS 102 filings produced by a single accounts-production stack, FINSTRUCT raises macro accuracy from 50.6% to 74.8% and micro accuracy from 63.1% to 85.1% over a deterministic parser. More importantly for deployment, it recovers structure at 83.5% accuracy on a filing where the rule-based parser silently misses whole sections, and its agreement signal separates more reliable predictions from cases requiring review. These results position logical structure recovery as a practical document-understanding layer for regulated financial automation, rather than a conventional parsing problem alone.

Agent skills extend language-model agents with task-specific procedures, scripts, and references, but the tasks and environments they target continually change. Existing methods improve skills in bounded runs and retain only the final artifact, discarding the decision history that later agents need to interpret prior revisions, evaluations, and rejected alternatives. We introduce SkillHone, a harness for continual agent skill evolution grounded in persistent decision history. SkillHone pairs skill revisions with evaluation-side evidence that supplies practice feedback, recording structured histories of diagnoses, revisions, evidence, and outcomes. Role-separated subagents run candidate skills on practice probes with redacted reporting and propose revisions informed by prior decisions, enabling cross-session refinement without rediscovering past rationale. On deep-research benchmarks, SkillHone runs without a pre-integrated search stack and outperforms the commercially backed deep-research agent by 15.8 points on GAIA and 3.2 points on WebWalkerQA-EN, while also exceeding prior skill-evolution methods. We further deploy SkillHone on internal tool-mediated analysis scenarios, where it improves accuracy by an average of 18.8 points across seven settings.

Click-through rate (CTR) prediction is a critical task for e-commerce advertising systems. However, industrial ranking models still rely on sparse IDs and historical behavior signals, which limits their generalization to cold-start products, long-tail queries, and complex user intents. Although large language models (LLMs) provide strong semantic understanding and can generate behavioral rationales, directly using them for online ranking is impractical under strict latency constraints.

We propose GRAM-V2, a rationale-supervised ranking framework that transfers LLM-generated behavioral rationales into an efficient CTR model. GRAM-V2 uses a Teacher LLM to generate click-oriented rationales, distills them into a lightweight Student LLM, and aligns a two-tower CTR model with the Student’s representations and rationale signals. During online serving, GRAM-V2 uses only distilled representations and cached item embeddings, without generating rationales autoregressively. Large-scale offline experiments and online A/B testing show that GRAM-V2 improves CTR prediction accuracy, CTR, and advertising revenue while satisfying production latency constraints.

Although Large Language Models (LLMs) offer significant advantages for creative tasks like game narrative design, their application remains largely confined to academic exploration. Existing methods often fail to integrate with specific game assets, production constraints, and iterative workflows essential for industrial development, limiting real deployment in professional studios. To bridge this gap, we present D2GS, a data-centric framework for LLM-based game script generation designed for industry. D2GS supports narrative designers end-to-end, from ideation to engine-ready scripts. It grounds creation in multi-source, game-specific data and mimics human iterative refinement. The framework includes four modules: (1) knowledge augmentation for multi-source data preparation; (2) logline and plot outline generation with a critic-feedback revision loop; (3) character and world enrichment using role-assigned LLM agents to simulate interactions; and (4) script rephrasing to ensure game-engine compatibility. Experiments show D2GS outperforms strong baselines in coherence, relevance, creativity, and overall writing quality, while staying faithful to predefined settings and assets. Industrial-scale validation further confirms its practicality for real-world narrative design pipelines.

Language models increasingly serve as advisory systems in maintenance operations. To prevent hallucination, established approaches ground these models in procedural documentation, constraining them to prescribed sequences. In practice, however, operators may stray from these steps, requiring models to recognise unscripted, off-procedure utterances. Current benchmarks rarely prioritise this capability. We introduce DiagFlowBench, a dataset of 50 industrial diagnostic flowcharts from a consumer manufacturer converted into 1,676 multi-turn conversations that contrast compliant with off-procedure utterances. Evaluating a panel of ten commercial and open-weight models reveals high variability in abstention rates, with models often selecting a real but contextually inadequate step rather than fabricating facts. The inherent plausibility and authority of this mapped but wrong advice exposes a challenging vulnerability for grounding systems.

AI text detectors are increasingly employed in academic settings, but it remains unclear whether their outputs reflect AI authorship itself or broader linguistic features associated with polished academic English. Previous studies have reported high false-positive rates (FPRs) for non-native English writing, but population-level comparisons confound authorship with differences in topic, domain, and writing style. Professional editing provides a useful setting for examining this issue because it changes the linguistic form of manuscripts while preserving authorship and content. We examined 135,389 document pairs from a professional English editing service (2018–2025), comprising non-native manuscripts and their native-edited versions, to assess how editing affects detector responses controlling for content and authorship. For the 13 AI text detectors, FPRs for human-written texts varied widely, from 0.0% to 100.0%. Responses varied across detectors: the same edits increased AI scores in some detectors but decreased them in others. Notably, score changes correlated with the extent of editing. The findings identify professional editing style as a key confounding variable in AI detector outputs, rather than establishing a full separation of text origin from linguistic style, raising concerns about fairness and reliability in academic settings.

Chain-of-thought (CoT) reasoning has become a default paradigm across reasoning-intensive tasks and is widely assumed to be, at worst, harmless. Recent efficient-reasoning work treats unnecessary CoT purely as a cost to be reduced. We show that in boundary-dense classification tasks such as content moderation and toxicity detection, CoT is instead actively harmful. We identify the overthinking effect: extended reasoning amplifies weak surface signals into false evidence of violation, inflating false positive rates on ambiguous non-violating samples while its gains on ambiguous violating samples mask the harm in aggregate metrics. To address this, we propose Oreki, an adaptive CoT framework that uses K-fold cross-probing to label per-sample reasoning necessity and fine-tunes on a mixture of full-reasoning and reasoning-skipped formats. Across four base models on video moderation and multi-label toxicity detection, Oreki preserves CoT’s recall gains while suppressing its false positive inflation, outperforming both Always-CoT and Never-CoT. On toxicity detection, it cuts low-agreement false positive rate from 0.467 to 0.257 while raising overall F1 from 0.792 to 0.851.

E-commerce platforms increasingly display clickable query suggestions alongside items in the user feed, enabling users to refine or expand their intent without manually reformulating queries. Existing approaches either mine suggestions from historical logs—limited to past behavior and blind to long-tail, personalized intents or rely on off-the-shelf LLMs whose lack of platform-specific knowledge yields fluent but generic queries disconnected from real click behavior. We propose EAGER (Enrich-and-Align Generative Query Recommendation), a two-stage framework for generating query suggestions from clicked items. In the enrichment stage, supervised fine-tuning (SFT) follows a four-stage curriculum that scales information richness (from item-only to user-conditioned) and reasoning depth (from direct to chain-of-thought). Each stage incorporates rationale augmentation, diversity regularization, and self-distillation. In the alignment stage, we post-train via GRPO with a hybrid reward of multiple rule-based business signals and a preference-aware click reward. Extensive offline experiments and online A/B test demonstrate the effectiveness of EAGER, which has been deployed in production at a major e-commerce platform.

Speech enhancement (SE) is commonly applied as a preprocessing step in spoken AI pipelines under the assumption that better audio quality improves downstream task performance. Whether SE-induced distortions propagate to downstream LLM task performance remains an open question. We introduce Output Divergence Rate (ODR), which measures how often SE changes an LLM’s intent classification relative to clean speech, and benchmark five conditions on 2,974 SLURP clips using Whisper large-v3 and wav2vec2-large cascades. Every condition produces ODR significantly above zero (p < 0.001, binomial test). MetricGAN+ more than doubles ODR versus unenhanced noisy speech (0.318 vs. 0.135) despite improving PESQ, and unmitigated echo reaches an ODR of 0.836 through speaker substitution, a failure WER cannot capture. Audio quality metrics range from near-zero to moderate correlation with ODR (SQUIM-MOS 𝜌\=\-0.068, PESQ 𝜌\=\-0.467). The MetricGAN+ and echo results replicate across ASR architectures, indicating that standard audio quality metrics are insufficient for LLM pipeline quality.

High-stakes tabular pipelines rely on scattered, static quality checks that are hard to evolve and audit against policy intent. Regulated domains particularly suffer as the LLM-era automation raises expectations for traceability and accountability. We propose a vision for an ontology-grounded, repository-native data quality layer in which rules are first-class artifacts: versioned, provenance-linked to governance texts, and re-playable through immutable execution traces. As a first step toward validating this vision, we present a concrete implementation of its agentic rule-discovery core with two rule acquisition modes (Full-Agentic and Guided) and a human-in-the-loop (HITL) review interface. We define an evaluation framework and run experiments addressing whether our fully autonomous, divide-and-conquer agentic rule discovery produces rules of the same quality and diversity as those specified by a human expert, reporting preliminary results on two real world industry datasets in the finance and retail domains.

LLM assistants deployed outside the United States often inherit US-centred interactional defaults. We introduce an Australian-facing assistant, and test whether local voice is better learned during fine-tuning or imposed with an inference-time prompt. We define Australian Voice as pragmatic alignment, directness, modesty, warmth, register control, and task fidelity, rather than slang or spelling substitution. We build a governed dataset from user research, human rewrites, contrastive negatives, reviewed synthetic examples, and replay data. In a blind evaluation by Australian English users (356 question-level ratings over 89 prompts), the raw 50% voice/replay mix scores best on all five dimensions (4.24/5; 102 best-pick wins). Prompted variants underperform every matched fine-tuned model.

Graph-based retrieval-augmented generation (RAG) is widely used for multimodal, cross-document question answering. However, building corpus-level graphs is expensive, slow to query, and difficult to maintain. We present TrioRAG, a graph-free multimodal framework that integrates evidence from three complementary signals: the question, the anchor image, and a VLM-enhanced query generated from both. Each signal retrieves independently over a shared multi-vector index of page text and page images, and the results are combined through late fusion. Further, we introduce AutoQA, a multimodal automotive benchmark whose questions are grounded in noisy, web-sourced images rather than clean document-sourced figures. Its questions require reasoning across manuals. We position it as a model-curated testbed rather than a human-validated gold standard. Across three benchmarks, TrioRAG matches or outperforms graph-based systems while reducing total cost and accelerating per-query inference by 1.6-2.3×. By construction, AutoQA grounds its questions in out-of-corpus web images. In this setting image retrieval reaches only 19.3% document-level recall, while text-derived signals, especially the VLM-enhanced query, keep retrieval robust.

Large-scale visual search platforms maintain production text-image embedding models trained on behaviorally-mined query-image pairs, where the text encoder is implicitly tuned to organisation-specific search vocabulary. Extending such a system to video retrieval is non-trivial: retraining risks disrupting hard-won query understanding, while naive temporal aggregation discards the sequential structure that distinguishes a door opening from a door closing. We add a lightweight temporal attention pooling head over independently encoded frame embeddings, keeping both encoders frozen and training only the pooling head and positional encodings. We train on a large-scale dataset from a commercial creative platform using bidirectional InfoNCE with a two-stage hard negative mining pipeline. Benchmarking against LoRA fine-tuned Gemma 3 (12B) and SmolVLM2, neither matches our approach despite substantially larger parameter counts. We also introduce an evaluation benchmark with graded relevance judgments annotated by a VLM judge, where each query is mapped to all relevant videos in a large corpus rather than a single ground-truth clip, avoiding protocols that greatly underestimate model quality. Our model improves nDCG@1 by 10.1% over mean-pooling overall and 25% on complex action queries. With Matryoshka Representation Learning, 256-dim embeddings retain 95.6% of full performance, and int8 quantization costs under 1% in retrieval quality.

Data-residency constraints force enterprises to self-host LLMs, but continuous adoption of newer models without decommissioning their predecessors expands the serving fleet, fragmenting a finite GPU pool. We consolidate traffic from over 200 internal applications onto a single model by closing quality gaps identified through production error analysis along three axes: instruction following, function-calling, and internal task distribution. Quality is tracked by offline benchmarks stratified to production traffic and scored by deterministic verifiers or calibrated LLM judges. Rather than optimising all objectives jointly, which introduces cross-domain reward interference, we train a separate GRPO expert per axis and merge them via two-stage SLERP. Each expert’s reward exposes a distinct failure mode, namely semantic collapse, over-calling, and verbosity hacking, each requiring a domain-specific fix. In non-reasoning mode the recipe surpasses a ∼7× larger by total parameters baseline on the in-house Arena with 69.6 to 65.8, instruction following with 0.85 to 0.83, and function-calling with 0.79 to 0.77, while lifting general dialogue benchmarks. The model absorbs 50% of platform traffic, 116M requests per month, at a fraction of the serving cost.

Extracting structured fields from hundreds of millions of documents annually remains costly in regulated industries: bespoke OCR cascades cover only a fraction of workflows, privacy rules preclude external models, and existing open-source VLMs that clear quality thresholds cost more to serve than human annotation. We present a deployed document-understanding system built on a Mixture-of-Experts VLM (35B total, 3B active), fine-tuned on in-house production data mixed with open-domain documents curated by a Difficulty-Aware pipeline for layout diversity, fact-extractability, and cross-model consistency. Fitting on a single H100 and serving heterogeneous workflows via prompting, the model leads all deployable (non-reasoning) baselines up to an order of magnitude larger. A quality-adjusted cost analysis, with confirmation and correction costs calibrated from production telemetry, shows it reduces expected costs by over 80% against the human baseline and by more than 50% against the best competing open-source model, while larger baselines remain economically unviable.

The detection of compromised business ad accounts is a critical challenge in digital advertising, as attackers exploit hijacked accounts to launch fraudulent campaigns. Large Language Model (LLM) agents show promise for integrity violations, but we find that they cause business friction through hallucinated mistakes on hard cases. Through a deployment study we observe that an autonomous LLM agent is a strong, recall-heavy signal extractor but an unreliable final arbiter for ambiguous compromise decisions, conceding substantial precision. We therefore pair the agent with a knowledge-based decision stage: the agent investigates an account and emits a structured, interpretable signal vector, and a neuro-symbolic stage makes the decision in three layers, symbolic rules discovered automatically with an Inductive Logic Programming (FOIL-IE) algorithm, a Naïve Bayes layer that calibrates those rules, and a data-tuned contradiction layer. We evaluate on a large oversampled general population (compromise oversampled, used because hard-case labels are scarce) and a realistic, low-prevalence production sample with subject-matter-expert (SME) labels, benchmarking against tree-ensemble baselines under identical conditions. The knowledge stage restores much of the precision lost by the agent (up to doubling it). On realistic low-prevalence data the full three-layer stage exceeds human expert reviewers and tree-ensemble baselines, with each layer contributing; its rules encode domain priors and generalize from few labels where ensembles overfit, while remaining interpretable and auditable.

We ask whether AI agents powered by locally deployed large language models can reliably automate expert-defined hardware design workflows in an industry-realistic tool-calling setting. In these environments, engineers issue repetitive, dependency-ordered operations—such as creating components, adding ports, and wiring connections—through specialised tools. Confidentiality constraints on component specifications and naming conventions often preclude hosted proprietary APIs, motivating the use of locally deployed models. To study this setting, we build a Model Context Protocol (MCP) server that reproduces the state and dependency logic of a proprietary hardware design tool used in embedded system development and construct a benchmark covering single-operation edits, multi-step dependency chains, invalid requests, misspelled prompts, and multi-server tool contexts. We evaluate seven open-source models comparing pipeline choices including system prompts, tool-description detail, context scope, and single-agent versus multi-agent architectures. Results show that strong models can achieve near-complete expected-call coverage on the benchmarked workflows, but reliability depends strongly on both task structure and agent configuration. Comprehensive tool descriptions consistently reduce failures, few-shot prompting can cause severe inaction for some models, cumulative context harms constrained models, and multi-agent decomposition helps weak workers or long sessions at the cost of additional calls. These findings provide practical guidance for deploying local LLM agents in stateful hardware design environments.-

Multilingual intent classification is central to logistics customer-service systems, where models must handle noisy user queries across languages and hierarchical labels. Yet many multilingual benchmarks rely on machine-translated text, which is cleaner than real customer requests and can overestimate robustness. We present a public benchmark for hierarchical multilingual intent classification from real logistics logs: approximately 30K de-identified, single-utterance queries curated from 600K records through filtering, LLM-assisted quality control, human privacy review, and sampled label auditing. The dataset uses a two-level taxonomy with 13 parent and 17 leaf intents; English, Spanish, and Arabic are seen languages, while Indonesian, Chinese, and Thai support zero-shot evaluation. To measure the gap between machine-translated and professionally localized evaluation, we provide paired test sets derived from the same source items. We benchmark multilingual encoders, embedding models, and small language models under flat and hierarchical protocols, and find that machine translation tests yield higher accuracy in most settings. [Prompts and sample data are available at <https://github.com/JTExpress-MICCS/MICCS>. A preprint version of this work is available at <https://arxiv.org/abs/2603.23172>.]

Translating a story renders its words into a new language; adapting it transposes its world into a new culture. Large Language Models (LLMs) translate and adapt individual passages effectively, but long-form novels spanning ∼100 chapters suffer model failures. We formalize long-form cross-cultural story adaptation as a generation task parameterized by source and target language-culture pairs, with three structural axes: A1 identity coherence across narrative arc, A2 cultural-system alignment within scenes, and A3 dependency-chain integrity among related entities. This is addressed through a five-stage agentic adaptation framework comprising entity registry, target-culture plan, dependency-aware batched localization, validation and rewrite. Evaluation is conducted through ablations across three language-culture pairs and two model families; the proposed system outperforms across all three axes, reducing identity errors by 65%, dependency errors by 64%, and cultural misalignment errors by 27% against competitive baselines. Further validation through a production A/B test on a commercial storytelling platform demonstrates listener retention comparable to that for human-adapted content, reflecting viability of the framework.

Vision-language models (VLMs) are increasingly used to extract structured fields from business documents, yet most evaluations report accuracy on clean benchmarks and offer little guidance to practitioners choosing an approach for a given task complexity. We address this gap with a measurement-grounded study and an open-source release. Across eleven systems (three commercial, two reasoning, five open-source VLMs in pretrained and fine-tuned form, and a non-LLM OCR→regex floor) scored on a 750-document held-out pool of synthetic checks, fine-tuning on 3K samples lifts the best open-source VLMs above F1 0.98—above every zero-shot commercial system on this task—while GPT-5 leads the commercial pool on F1 and Claude Sonnet 4.5 collapses on Date. To turn these measurements into actionable choices, we introduce a practitioner-oriented selection framework that maps a task profile (quality, latency, governance, volume) to a recommended approach via filtering and total-cost minimization, illustrated on a hypothetical mid-volume document-extraction scenario.

Structured information extraction at small-language-model scale is a routine production workload, yet each new task added to a serving stack can demand new task-specific model state. We propose a schema-hierarchical soft prompt: a small learned prefix, trained on a frozen backbone, that gives every JSON-schema node its own slot, orders the slots by the schema, and marks tree depth with learned embeddings. Used alone on the open USPTO-ORD patent task, it recovers most of a LoRA reference’s extraction quality while training more than an order of magnitude fewer parameters, and ablations attribute the gain to depth and schema order rather than token count. Used with a LoRA adapter on an internal booking task, the same slots stabilize one of two held-out categories, so we report this as a bounded mechanism study rather than a general gain. Inference-time slot replacement indicates that position supplies the schema scaffold while learned content stabilizes generation. Measured serving gives a far smaller task artifact and higher throughput than multi-LoRA as interleaved tasks grow. The soft-prompt module is available at https://github.com/microsoft/Hierarchical-Soft-Prompt.

Large language models (LLMs) struggle to classify text into taxonomies with many semantically similar labels, as the distinctions are domain-specific and not captured by pre-training. To handle large label spaces, a common approach retrieves top-K candidate labels by embedding similarity and prompt the LLM to choose among them. However, top-K retrieval reduces the number of candidates but does not help the model tell similar ones apart. When two similar labels both appear as candidates, the model lacks the signal to choose correctly between them. We propose a framework that (1) identifies which label pairs the model struggles to distinguish, (2) expands the candidate set to include confusable labels, and (3) generates targeted rules to differentiate between similar candidates. The framework requires no fine-tuning, and the generated rules transfer to smaller, cheaper models. On three benchmarks (WOS, Flipkart, LEDGAR), our approach improves Macro F1 by up to 10.0pp over retrieval baselines, with smaller models (2B–20B) gaining up to 11.5pp via cross-model transfer.

RLHF, RLAIF, and RLVR align LLMs with human instructions but rely on costly human labels, potentially biased LLM judges, or task-specific verifiers. We propose PolyMetric Alignment (RLMA), a reference-based reward framework for supervised-answer tasks that fuses heterogeneous evaluation metrics against a given reference answer using a Residual Set Transformer (RST) trained with a Bradley-Terry objective. Empirically, RLMA is stronger than all three paradigms, achieving 98.0% of the Pass@K oracle on a 9-pool benchmark, with gains of +17.6 pp over the RLA single-metric baseline and +7.4 to +15.4 pp over RLHF, RLAIF, and RLVR. RLMA also generalizes broadly, reaching 97.2% of oracle on unseen GSM8K and, when used as a DPO reward, improving TriviaQA Pass@1 by +4.80 pp on Qwen-2.5-7B-Instruct. Finally, RLMA is efficient to deploy. Its 113K-parameter gate requires only 0.33 ms per candidate, roughly 300x faster than a single 8B reward-model forward.

Currently, most agentic systems are driven by mainstream closed-source large language models (LLMs), such as GPT-5 and Gemini 3, resulting in high operational costs. As a result, deploying customized small language models (SLMs) to replace LLMs in agent components has attracted increasing attention. However, the limited capabilities of SLMs often render them inadequate for agent tasks. To address this, we propose Edit4Agency, a novel framework that utilizes model editing to efficiently and effectively enhance agent sub-capabilities of SLMs. By formalizing tasks as a sequence generation problem triggered by specific conditions, the framework embeds task triggers into system prompts and performs model editing through localized parameter updates, eliminating the need for large-scale retraining. Extensive experiments across multiple datasets and case studies in agent systems demonstrate that model editing, with only 50 data samples, enables a base model without instruction tuning to acquire new skills such as tool calling while maintaining its general capabilities. The results show that Edit4Agency outperforms instruction-tuned SLMs, constrained decoding, and SLMs trained with large-scale supervised fine-tuning or reinforcement learning methods.

Large language models (LLMs) are increasingly considered for safety-critical engineering, yet their reliability in regulated functional-safety workflows remains underexplored. We introduce SAFARI (Safety-Aware Functional Automotive Risk Inference), the first industrial benchmark for LLM-assisted automotive Hazard Analysis and Risk Assessment (HARA) under ISO 26262. It contains 3,000 de-identified industrial HARA cases and evaluates two coupled tasks: open-ended hazard analysis and standards-grounded risk assessment. To evaluate open-ended HARA artifacts, we propose the first reference-anchored LLM-as-a-judge protocol with high expert correlation. Experiments with nine frontier LLMs show that models often produce plausible hazard narratives but remain weak at ISO 26262 risk classification, with the best ASIL macro-F1 reaching only 0.261. Chain-of-Thought prompting provides limited benefit and often degrades categorical risk assessment. Error analysis further localizes major failures to scenario-critical context omissions during hazard generation and to controllability misjudgments during risk assessment, indicating where expert oversight should be concentrated. The dataset can be obtained from https://github.com/xixi47520-hash/HARA.

LLM agents routinely spend orders of magnitude more tokens on context than on generation, re-sending interaction history, tool schemas, and system prompts at every turn. We introduce **SCOPE**, a unified framework for *context compilation* that compresses these three strata jointly under a fixed token budget while preserving agent behavior. SCOPE combines: 1. A learned *minimal sufficient context* reranker for memory, trained via leave-one-out supervision; 2. A plan-conditioned tool compressor that speculatively loads likely tools while retaining lightweight signatures for the rest; 3. A constraint-aware prompt compressor that preserves instructions. A lightweight planner coordinates these components via a contextual bandit, allocating budget across memory, tools, and prompt to maximize fidelity to full-context execution. On 3142 chat trajectories, SCOPE achieves **96.8%** decision fidelity at **4.46×** compression (60K tokens/turn), matching full-context task success within **0.9%** while using only **22%** of the tokens. Across operating regimes, SCOPE Pareto-dominates strong single-stratum baselines and naive compositions, with gains exceeding **11%** at matched cost. Our results show that optimizing context in isolation is insufficient: coordinated, budget-aware compression is key to scaling LLM agents without sacrificing reliability.

LLM-based factuality judges provide scalable evaluation signals, but their metrics are often systematically biased relative to human judgments. We study human-anchored factuality evaluation under limited annotation budgets, where judge predictions on the full dataset are combined with human labels on a small selectively sampled subset to obtain statistically valid estimates. The efficiency of this approach depends critically on which examples receive human annotation: in factuality evaluation, judge-human misalignment is not driven solely by low confidence, but also by structured failure modes such as incomplete evidence, temporal mismatch, unverifiable claims, and rubric misalignment. To exploit this structure, we introduce a factuality-specific annotation policy design pipeline that uses failure-space analysis (FSA) to derive diverse predictive signals for modeling human-judge misalignment. On an internal reference-based factuality evaluation system (AutoFA) and RAGTruth, where judge-predicted estimates substantially underestimate human-judged factual accuracy, our FSA-guided policy improves annotation efficiency over uniform sampling and uncertainty-driven baselines, achieving effective-sample-size gains of 40.3% on AutoFA and 27.1% on RAGTruth.

Adverse drug reaction (ADR) severity prediction is a key task in pharmacovigilance and post-market drug safety surveillance, but is of- ten challenged by noisy and heterogeneous clinical data. We propose a retrieval-augmented learning framework that leverages historical FAERS reports to construct evidence-enriched training instances for ADR severity prediction. Each ADR query, including patient demographics, drug exposure, and indications, is encoded using a fine-tuned BGE-M3 embedding model. We retrieve the top five relevant cases using BM25 lexical search and the top five semantically similar cases using dense vector search, and fuse them via an attention-based aggregation mechanism to produce retrieval-aware representations. These representations are evaluated using machine learning, deep learning, transformer, and fine-tuned large language model approaches for binary severity prediction. Results show that a tree-based ensemble achieves the best performance with an AUC of 82.3%, outperforming the strongest deep learning baseline by 2.3 percentage points and fine-tuned LLMs by 17 percentage points. Overall, the findings demonstrate that retrieval-enhanced data preprocessing can improve representation quality for ADR severity prediction, while the proposed upstream data engineering approach enables classical ensemble methods to remain highly effective for large-scale pharmacovigilance risk prediction.

Agent ecosystems now include thousands of MATS components (Models, Agents, Tools, and Skills), yet their discovery still relies on in-context routing. These systems read a registry (names, hints, or descriptions, as context budget permits), pick a candidate, invoke it, and retry on failure. This pattern degrades with scale, and registries are growing fast. We recast capability discovery as search over a registry by defining an offline enrichment step that turns sparse metadata into searchable profiles, and an online retrieve-then-rank pipeline that returns a ranked shortlist without invoking any candidates online. We show that from N=10 to 7,278 capabilities, in-context routing’s top-1 accuracy (Match@1) collapses (0.85 to 0.12), while retrieve-then-rank degrades more gently (0.81 to 0.39) because its reranker still ranks the right capability first 0.70-0.87 of the time once retrieval finds it. In the Nova Micro sweep, the crossover is around N=500. We compare against two in-context baselines. Full-Ctx puts the whole registry in the prompt and asks the LLM to pick. Search Pick gives the LLM a search tool to narrow candidates before it picks. At full scale the pipeline leads Search Pick by 6.5 percentage points (pp) on Match@1 at about half the cost. It reduces cost 70x versus Full-Ctx. We use a fixed configuration (same enrichment, retriever, and scorer weights) across agent, tool, and skill registries. The pipeline runs in production as the default capability-discovery layer of a large-scale multi-agent platform.

Hi, I’m Gisela, GLS’s assistant. How can I help you today? This is how the Spanish deployment of the GLS Chatbot greets its customers. Behind the greeting is a governed multi-agent system for parcel-logistics customer service, built on self-hosted open-source small language models with RAG. An orchestrator separates routing, execution, and generation; probabilistic components are bracketed by deterministic safeguards, customer-impacting actions require explicit user confirmation, and out-of-scope, abusive, or manipulative inputs are refused. In production since November 2025, the system has handled over 1.7 million conversations across Spain and Germany. In Spain, the average realised cost during May 2026 was approximately 0.07€ per conversation, with sub-3-second latency. We describe the architecture, evaluate the Spanish deployment on task completion, routing conformance, and RAG answer quality, and report lessons from production.

Machine learning practitioners routinely rely on public code repositories accompanying research papers, libraries, and open-source projects to evaluate, baseline, or build upon prior work. Yet getting such repositories to run and reproduce results often involves non-trivial effort, and is especially challenging when the target software and hardware stack differs from the one the original code was developed on. A prominent instance is running repositories originally built for CUDA-enabled hardware on AMD GPUs supporting the ROCm software stack. Here, successful installation alone is not a sufficient signal: a range of subtler issues may cause the code to either fail outright or silently fall back to inefficient execution. We present Repo2ROCm, an agentic system that migrates CUDA-centric ML repositories to ROCm in an automated manner. Repo2ROCm reconstructs a ROCm-compatible environment, adapts hardware-specific assumptions, executes a real workload, and confirms GPU-backed execution emitting auditable artifacts at every step (environment specs, source patches, execution logs, and verification evidence). We evaluate on ResearchBench, 60 repositories belonging to recent papers from top AI conferences. While only 18/60 run out of the box, Repo2ROCm reaches verified functional reproduction for 59/60 (3.3× improvement). Against popular general-purpose coding agents on a curated ten-repository diagnostic set (with all agents using the same Claude Sonnet 4 backbone), Repo2ROCm achieves the highest Portability Fidelity Score (PFS), a measure of how faithfully a run preserves the accelerator-critical execution path, real workload, and verified environment, scoring 77.3 vs. 61.4 (Claude Code), 58.7 (Mini-SWE-Agent), and 40.3 (Cursor Agent).

Root Cause Analysis (RCA) for defects in large e-commerce enterprises is manual and slow: a single defect spanning thousands of microservices and SOPs takes experts one to two weeks to diagnose. Generic agentic RCA frameworks fail on this workload because they stop at proximate causes, cannot verify numeric claims, ignore past reviewer decisions, and return enumerated hypotheses rather than the quantified narratives business teams act on. We present Trace, an end-to-end agentic RCA system that combines supervised hypothesis-driven investigation with recursive why-drilling, an evidence-traceable multi-tool suite with stable artifact identifiers, a feedback retrieval and ingestion pipeline, and a two-layer confidence framework combining six mechanistic checks with an LLM judge - including a novel numerical-traceability score. Across several thousand cohorts spanning eight use cases in three metric categories (cost, quality, and inventory), Trace achieves 65.4% expert acceptance; on a 240-cohort head-to-head benchmark it reaches 67.1% vs. 38.8% for a matched ReAct baseline (+28.3pp, p<0.001) with calibrated confidence (AUC 0.819, ECE 0.041).

External memory banks can improve test-time reasoning in frozen large language models without updating backbone parameters. Yet similarity-based retrievers and pointwise selectors score memories independently, ignoring interactions and presentation order. We introduce , an LM-style model for reinforced memory selection that casts context construction as autoregressive index decoding. From a densely retrieved candidate pool, a lightweight Transformer encoder–decoder generates an ordered index sequence [i₁,…,i_k], modeling dependencies among memories while leaving the backbone frozen. This formulation supports standard language-model training. For sparse binary verifiable rewards, we further propose , a confidence-aware extension of group-relative policy optimization that promotes correct, low-confidence rollouts, penalizes confidently incorrect ones, and retains learning signals in reward-homogeneous groups. Across three mathematical reasoning and two reasoning-intensive question-answering benchmarks with three Qwen backbones,  achieves an average score of 84.9 on Qwen-plus, exceeding the strongest memory-augmented baseline by 2.3 points.

Full-duplex spoken dialogue models aim to make voice agents more natural by allowing them to listen, speak, pause, and respond during ongoing conversation. However, it is not clear whether full-duplex benchmarks behave the same way when models are evaluated in a different language. To investigate this, we introduce a French full-duplex benchmark (FDB) with two variants, CALLFC-FDB for Canadian French and MEDIA-FDB for European French, and compare them with an English FDB. Built from real spoken resources, these benchmarks evaluate key full-duplex skills, including pause handling, turn-taking, backchannels, and interruptions. Beyond introducing French FDBs, we evaluate human–human conversations with FDB metrics to better understand the values these metrics take in real-world dialogue. Our analysis reveals that most timing-based metrics behave similarly across languages, while content-based evaluation degrades under language mismatch. We also find a trade-off between optimizing benchmark metrics and preserving conversational naturalness.

Production text-to-code systems increasingly use LLMs to translate natural-language analytics requests into executable SQL and pandas code, serving both business users, who describe desired results in application terms, and technical users, who name tables and columns directly. Existing Text-to-SQL and Text-to-Python benchmarks provide coverage, but they do not isolate when SQL and pandas performance diverge. We introduce BIRD-Persona, a persona-stratified extension of BIRD-Python that adds novice/expert rewrites (business-style and technical-style phrasing) to tasks with matched SQL and pandas ground truth, plus schema variants and task-structure annotations. Across 1,534 tasks, nine models, and 245,592 evaluations, BIRD-Persona reproduces the previously reported SQL-over-Python gap: SQL outperforms pandas for eight of nine systems. The gap is not uniform: expert, schema-explicit prompts improve both paradigms but help SQL more, schema representations have mixed model-dependent effects, and selected-column burden widens the gap while subqueries do not. By making user phrasing a controlled benchmark dimension, BIRD-Persona provides a foundation for persona-aware and paradigm-aware evaluation of natural-language analytics systems.

For emerging scientific research domains, local Small Language Models (SLMs) are becoming more attractive, as they offer stronger privacy control and more stable deployment pipelines than Large Language Models. However, in practice, scientific question-answering on SLMs often operates under inevitable constraints: small literature collections, fragmented evidence, limited context window and reasoning abilities. We propose the Evidence-Grounded Typed Knowledge Graph (EGT-KG), a retrieval framework to improve information retrieval with local SLMs. We assessed three question-answering settings: a vanilla Retrieval-Augmented Generation (RAG) workflow and two EGT-KG workflows: an automatically generated relation schema (AS) and an expert-defined relation schema (ES). Our experiments were evaluated with a six-dimensional evaluation framework (S3CRF: Soundness, Correctness, Completeness, Conciseness, Relevance, Fluency) on a Biopolymer-bound Soil Composite literature benchmark, showing that EGT-KG outperforms the vanilla RAG method in most settings, with the best improvement from llama3:8b: a Final Score of 70.37 (+14.67%) and 68.82 (+12.14%) by AS/ES EGT-KG variants.

Contrastive fine-tuning of a single embedding model across heterogeneous retrieval tasks is challenging when the tasks differ in label structure. In product-compliance retrieval, some tasks have multi-label supervision — every item carries a set of compliance categories, and many in-batch items share at least one category with the query, producing frequent in-batch false negatives. Other tasks (e.g., cross-catalog product matching) have no category labels: each query has one canonical match. Existing multi-label contrastive methods require every batch item to carry comparable labels, so they cannot serve a single embedding model trained on both label-bearing and label-free tasks. We propose label-aware false-negative exclusion: a zero-cost binary mask on the InfoNCE denominator that drops same-label candidates from the negative set rather than including them to positives. The mask has no effect on label-free items, so the same loss handles both task types. A 0.6 B-parameter embedding model fine-tuned this way handles four retrieval tasks simultaneously, lifting average Recall by 49% (0.489 → 0.729) and matching the 13× larger off-the-shelf Qwen3-Embedding-8B at 32× lower embedding dimensionality. Fine-tuning further enables learned query fusion: concatenating regulatory text with an example product, which hurts retrieval before fine-tuning, becomes the strongest query type after, beating either alone by 6-11%.

Vision-Language Models (VLMs) are increasingly used for long-document processing, where the inputs combine text with charts, tables, figures, and complex layouts. Deploying them means choosing how to feed the document to the model, which retriever to use when only a subset of pages is sent, and whether to run the model agentically or as a static pipeline. We study these choices on two long-document QA benchmarks with both frontier API and open-weight VLMs. First, on MMLongBench-Doc our six-tool agent with page, table, figure, and search calls pays off only once the answering VLM is large enough: with Qwen3.5-4B and 9B it trails static page input, with Qwen3.5-27B it draws level, and with Sonnet 4.5 it leads. On LongDocURL it is level with or ahead of static input at every reader. Its lead over the strongest static pipeline is clearest with the frontier reader on MMLongBench-Doc and narrows to within noise on LongDocURL. Second, retrieval modality matters more than the specific retriever: the strongest image retriever leads the strongest text pipeline, and on the text side a single off-the-shelf cross-encoder rerank essentially matches a much heavier multi-stage LLM pipeline. Top-k image retrieval is also the most token-efficient input at every reader we paired it with, at roughly a seventh to a quarter of the tokens of sending every page. Third, cutting across all three choices, three of our strongest pipelines succeed on different questions, and an oracle that picks the best pipeline per question gains roughly thirteen points over the best single pipeline, though evidence-type routing recovers almost none of it.

Streaming Automatic Speech Recognition (ASR), also known as online or real-time ASR, transcribes speech incrementally while audio is being spoken and is critical for many industry applications. Despite the strong performance of modern offline ASR models, streaming deployment remains challenging because limited future context can cause early recognition errors to propagate across later chunks. We define the task of streaming ASR error correction and propose StreamCorrect, a modular framework that fine-tunes a lightweight error corrector from self-generated streaming ASR errors and integrates it with existing streaming ASR systems. To enable training without human annotation, StreamCorrect introduces a timestamp-free boundary recovery procedure based on Prefix-to-Reference Alignment for generating chunk-level correction data. Across multiple languages and ASR backbones, StreamCorrect achieves up to 39% relative character error rate (CER) reduction with minimal latency overhead. Error propagation analysis further shows increasing benefits over time, with up to 7.2% absolute CER reduction in later audio chunks. StreamCorrect bridges offline ASR models and streaming deployment through a lightweight correction plug-in while preserving the pretrained ASR component without additional tuning or distillation.

A popular way to improve Retrieval-Augmented Generation (RAG) is to rewrite the user’s question into several variants and search with all of them. We test whether this actually helps once the underlying search is already strong. Under one fixed, competitive pipeline (BGE dense retrieval, cross-encoder reranking, and MMR diversification), we compare four query-rewriting strategies (S1–S4) against two strong LLM baselines (HyDE, Query2Doc) on three datasets (HotpotQA, AmbigNQ, and the 512K-document EnterpriseRAG-Bench) over three seeds with paired-bootstrap significance tests.

Our headline result is that rewriting alone is at best competitive with a strong baseline, but combining methods yields outsized gains because different strategies fail on different questions. A post-hoc union of four methods (S1+S3+S4+HyDE; seed-42 analysis) improves HIT@10 over the dense baseline by +12.5 points on enterprise-style data (51.70 vs 39.22), and a five-method union reaches 52.98 (+13.8 over baseline). Budget-matched controls (TopK-100 reranking, multi-pass HNSW) capture only ∼40% of this gain, confirming that complementarity, not retrieval budget, is the primary driver. On HotpotQA the union adds +1.6 to +1.8 points (p < 0.001), saturating the all-method oracle; on AmbigNQ the same fusion hurts (-2.4 below the best solo, p < 0.001), and we analyze when and why. Because rewriting is expensive (one LLM call each), we turn this into a cost-aware heuristic evaluated in simulation: a confidence-gated router that runs rewriting only when the baseline’s own top-1 score is low. It captures about half of the enterprise full-merge gain (+4.3 HIT@10 over the baseline) while paying the rewriting cost on <40% of queries, and automatically declines to rewrite on AmbigNQ, sidestepping the loss that blind merging causes. A downstream answer-quality evaluation confirms that the router improves F1 by +1.92 (p < 0.01) at roughly 40% of the expansion cost. In short: treat query rewriting as a complementary coverage source, applied through cost-aware routing, not as a standalone replacement for a strong baseline.

Large Language Models (LLMs) are increasingly deployed in high-stakes clinical applications in India. Speakers of Indian languages frequently communicate using romanized text rather than native scripts, yet existing research rarely quantifies or evaluates this orthographic variation in real-world applications. We investigate how romanization impacts the reliability of LLMs in a critical domain: maternal and newborn healthcare triage. We benchmark leading LLMs on a real-world dataset of user-generated health queries spanning five Indian languages and Nepali. Our results reveal consistent degradation in performance for romanized messages, with gap reaching up to 24 points across languages and models. We propose and evaluate an Uncertainty-based Selective Routing method to narrow this script gap. Our findings highlight a critical safety blind spot in deploying LLMs for health systems: models that appear to understand romanized input may still fail to act on it reliably.

When the key-value (KV) cache fills, a serving engine must choose which request to evict. We present PreemptBench, comparing seven preemption-target policies across seven reasoning-model families and two benchmarks. Our service-level objective (SLO) breach rate counts every accepted request, including late completions, post-acceptance failures, and unfinished work; this survivorship correction prevents deferred backlog from being omitted from the metric. In the low-breach regime, where last-in, first-out (LIFO) breach is below 15%, young-target-biased policies form a lower-breach cluster than policies without this bias, including random and largest-KV, consistent with a stable incumbent cohort; graded eviction depth is secondary. Within a pure remaining-decode rule, replacing the online predictor with a privileged empirical statistical length oracle, constructed from prior temperature-zero runs, yields no significant gain; neither significantly outperforms LIFO. It summarizes a prompt-conditioned length distribution, not the current execution’s realization, and is not SLO-optimal. Finally, the uncensored single-stream completion-time tail defines a conceptual feasibility floor, but our traces provide only a censored low-load proxy. Post-hoc rescoring across 600–1800-second deadlines finds aggregate memory-freeing reversals at tighter SLOs, but none at 1800 seconds. Thus LIFO is a strong conditional baseline, not a universal optimum.

We study postmortem remediation planning for large-scale cloud incidents, focusing on whether generated actions are feasible, safe, and executable under real operational constraints. While LLM-based systems can produce structured corrective actions, we find that 25–35% of generated plans across 8,200 real-world incidents violate at least one feasibility constraint due to hidden dependencies, policy limits, or risk budgets.

We introduce Solver-Verified Remediation Planning (SVRP), a decision-centric framework that separates action proposal from feasibility verification. An LLM proposes candidate actions, which are verified using an explicit constraint system and satisfiability solving; minimal unsatisfiable cores guide targeted repair or abstention. SVRP raises feasibility from 62.4% to over 93.1%, reduces unsafe actions by more than 50%, and improves expert-rated executability from 3.7 to 4.7, while preserving semantic intent. These results suggest that remediation failures often reflect structural constraints that benefit from solver-backed verification in addition to improved prompting.

Accounting fraud costs organizations billions annually, yet auditors rarely confront it as a single suspicious posting. Instead, they investigate coordinated *fraud schemes* such as fictitious vendor payments, ghost-employee payroll, and inflated revenues that span many journal entries, each individually routine, and only become visible when understood against the underlying accounting process. Existing AI systems either score isolated entries as anomalies or mine company-level disclosures; none evaluate whether a large language model (LLM) agent can query a live ledger, recover coordinated fraud patterns, and assign the correct scheme type. We release **ForensicBench**: the first benchmark for evaluating agentic LLMs on *scheme-level* journal-entry fraud detection. The dataset, **Forensic Ledger**, extends DataSynth with five injected fraud scheme types and scheme-level labels linking journal entries to coordinated instances. We evaluate twelve open-weight models under a single reference agent scaffold on a public leaderboard: among them, the best reaches only 34.7% Entry-F1, and Type-F1 is up to 13 points lower. We publicly release the dataset, evaluation harness, and a reference agentic baseline

Multimodal models that interpret screen recordings are usually fed raw sampled video frames,, a faithful but costly perception layer: each frame is a large image input, and dense coverage multiplies token usage. We ask whether a cheap, text-first index (on-screen OCR text plus per-state change descriptions, built once by a small local pipeline) is a better-value perception layer for the same model. On GUI-World, a released benchmark of GUI recordings with human-annotated multiple-choice questions and an objective exact-match metric, we compare the two in a controlled setting (same model, same questions), and replicate across two flagship models. On Claude Opus 4.8, answering from the index reaches 0.850 accuracy vs.0.902 for an 8-frame baseline (n=286) at 44.6% lower cost, and a lossless content compression cuts cost to 75.6% below frames with no measurable accuracy loss. On GPT-5.5 the trade-off is even stronger: the compressed index trails frames by only about 3% (0.874 vs.0.895) while using 82.8% fewer input tokens. The index is also stable across models, the compressed result differing by only about three points between them (0.843 on Opus 4.8, 0.874 on GPT-5.5). The honest summary is comparable accuracy at a fraction of the cost.

Production data integration is bottlenecked by repeated, lossy handoffs between data owners, engineers, and analysts who must collaboratively discover, structure, and query enterprise data. We present **D**ata **I**ntelligence **A**gents (DIA), a system of three agents (*Data Interpreter*, *Schema Creator*, and *Query Generator*) that compresses this workflow by treating autonomous coding agents (ACAs) as a first-class abstraction: rather than emitting text, the agents generate, execute, validate, and repair concrete artifacts, draw on a shared memory for experience reuse, and surface each for review by domain experts. DIA is deployed in production for enterprise customers. We study the *Query Generator* in depth and evaluate it in fully autonomous mode across seven SQL benchmarks spanning four task categories and four dialects. It matches or surpasses the best published results on all seven, demonstrating that an architecture grounded in execution, built on ACAs and a shared memory, generalizes across the data intelligence workload with adaptation confined to natural-language instructions.

Retrieval-augmented generation (RAG) is hard to monitor in production: exhaustive relevance labels do not exist for non-stationary multi-million-passage corpora that re-index in real time. As a result, retrieval quality is generally understudied and often deprioritised in favour of generation-oriented metrics. In this work, we propose auditing retrieval coverage by probing for evidence of missing documents rather than enumerating every relevant one. Our method Re:CAP (REtrieval Coverage Audit by iterative Probing) is a reference-free audit loop applied to a deployed RAG pipeline’s initial answer and retrieved context: it identifies the topics already covered, generates probing questions for plausibly missing topics, retrieves candidate documents, and applies an LLM-as-judge to retain only those that introduce previously-unretrieved information. On four public benchmarks, Re:CAP recovers 9–29% of gold labels that flat BM25 top-500 cannot reach, rising to 48% on TREC-COVID. On MuSiQue Re:CAP beats flat hybrid top-500 by +12.9 pp on recall at less than half the document budget. An ensemble BM25, dense, and hybrid baseline (top-500 each) still leaves out 21.2% of gold docs on TREC-COVID that Re:CAP recovers; human annotators judge that 78.9% of those structurally distinct documents add new information to the baseline answer (Fleiss κ = 0.79, n = 123), and 73.9% on live production traffic (n = 180). End-to-end recall is reproducible to within ±1% across three independent runs, making Re:CAP a stable instrument for periodic retrieval audits.

Multi-party financial chatrooms are vital for sales-and-trading professionals, but their complexity makes manual recovery of missed trades infeasible: each Request for Quote (RFQ) is an event whose final price and trade outcome appear many messages after the RFQ-trigger message (the inquiry message), interleaved with concurrent RFQs from other participants. We cast this as event extraction (EE) over multi-party dialogue and present FinDialogLens, a hybrid LLM pipeline in which compact fine-tuned classifiers act as inference-time scaffolds: they detect RFQ-triggers and price/trade outcome metadata, an RFQ-Level Module segments per-event RFQ windows, and a Trade Engine fills argument roles. With GPT-4o, FinDialogLens reaches 92.1% and 94.3% accuracy on final price and trade outcome, respectively, outperforming full-chatroom CoT prompting methods; fine-tuned open-source LLMs with as few as 3B parameters achieve comparable performance with modest in-domain data. To make the LLM-based solution practical at scale, a difficulty-aware router balances cost and accuracy by allocating RFQs between a low-cost rule-based engine and the higher-performing LLM-powered Trade Engine, cutting LLM calls by 85% on final price while recovering half of the accuracy gap to FinDialogLens (GPT-4o), saving over 300/day at our 70,000-RFQ/day scale.

Legal documents frequently contain heterogeneous references to judicial precedents, making automated Precedent Reference Extraction (PRE) challenging for legal AI systems. Existing approaches rely heavily on LLM-based pipelines, which are often effective but also incur recurring prompting costs in high-volume production workloads and struggle with tracing extracted entities back to their character offsets in the source text. In light of these issues, we present Notarius, a domain-specific and production-ready hybrid framework that decomposes PRE into two stages: Overlapped NER Feature Extraction (ONFE), which extracts fine-grained precedent features, and Precedent Feature Aggregation (PFA), which reconstructs complete references for downstream applications. Our implementation integrates the Unified NER method W²NER with domain-specific heuristics. We evaluate Notarius at both feature and reference levels on a collection of judicial decisions and compare it in production against an LLM-based pipeline. Notarius achieves higher extraction quality, eliminates recurring LLM API costs for PRE, and provides span-level offset traceability. The system currently powers precedent extraction on Jusbrasil, demonstrating that discriminative, domain-specialized NER architectures remain valuable for cost-effective, high-precision legal information extraction in the LLM era.

Conversational assistants can blend retrieval, action selection, escalation, and wording in one model path, or separate those roles. We report a production migration of a conversational assistant for customer support at a large accommodation marketplace (millions of conversations per month, 11 languages, 10-second P90). Dynamic Response (DR) replaces a single Qwen3-235B-A22B blended responder with a bounded ReAct orchestrator over typed tools plus a smaller generator that writes from a backend-validated context contract. Because the migration also changed prompts, alignment, and serving, we attribute each effect to its cause and claim as architecture effects only those measured on identical replayed turns: typed entity selection moves the reservation selector to a precision-first operating point (precision 8.3% to 89.1% at recall 75.2% to 67.3%), and typed action IDs with a membership check remove observed structured-action hallucination (2.14% to 0.0%). A low-ramp A/B test reproduces the replay escalation reductions: hard-escalation responses fall 5.60% to 3.08% and soft-escalation responses 9.56% to 2.49% while production handoff volume holds roughly steady, and self-solve is directional (+5.1 points, 95% CI [-2, +12]). Serving optimizations cut orchestrator P90 from 3.87s to 2.24s on a GPU footprint reduced by roughly one-third, and self-hosting reduces estimated annual model-serving cost by more than an order of magnitude.

Production LLM agents often waste latency and reliability by regenerating code for the same procedural steps on every request. We replace this inference-time coding loop with an agentic tool-making pipeline that compiles repeated SOP steps into validated, versioned tools before deployment. The tool-maker grounds synthesis in the live environment as it collects execution traces, observes backend schemas and values, generates candidate tools, and repairs them against labeled cases. At runtime, the production agent calls these tools directly and falls back to code generation only when needed. We deploy the approach in a Fulfillment Center alarm-triage system, where an agent diagnoses alarms against a 44-node SOP over heterogeneous metric backends. In production, tool calls reduce p50 latency by 42%. On 1500 historical alarms, they reduce end-to-end error rate by up to 53% by suppressing run-to-run variance in repeated steps. Because tools return compact structured verdicts, they also enable a simpler direct-call architecture, reducing p50 latency by a further 62% in a controlled ablation. Versioned tools also improve auditability and expose specification gaps and upstream data drift. Our results show that agents that build and maintain their own tool libraries, a key element of self-evolving agents, can make industrial LLM systems faster, more reliable, and easier to operate.

We present PLSQLBench, to our knowledge the first benchmark for evaluating whether LLMs can write executable PL/SQL programs, with correctness measured through execution-based tests. Existing LLM evaluations largely target general-purpose code generation or declarative text-to-SQL, leaving procedural database programming underexplored. PLSQLBench contains 2,865 instances: 2,594 single-turn tasks and 271 multi-turn conversations spanning 978 turns. The benchmark combines complex schema-grounded tasks over enterprise-style Spider 2 databases, simpler schema-grounded tasks derived from Spider, and MBPP-derived procedural problems, covering varying levels of database grounding and procedural complexity. Experiments with eight LLMs reveal recurring difficulties in schema grounding, PL/SQL dialect fidelity, procedural control flow, exception handling, and cross-turn consistency. Tool-augmented LLM agents improve performance on several schema-grounded evaluations, although substantial gaps remain. These results highlight procedural database-programming capabilities not directly assessed by conventional code-generation or text-to-SQL benchmarks. Our code is available at https://github.com/oracle-samples/plsqlbench.

Enterprise workflow automation typically requires retrieving a complete and accurate set of relevant information from large operational document corpora (SOPs, policy documents) given a specific scenario or request. In customer support (CS), for example, human specialists must determine the correct sequence of resolution steps for each case, a task that demands precise grounding in policy documentation. Meeting this bar in production typically requires complex document chunking, indexing, and retrieval pipelines, often layered with learned re-rankers, query rewriters, and bespoke agentic RAG infrastructure.

We present a simpler alternative: a portable skill consisting of a natural-language instruction, lightweight tools (a retrieval API, format-checking scripts), and a corpus enriched with an LLM-generated navigation layer, which is a structured "Wikipedia-style" index over the underlying documents. This agentic RAG approach is straightforward to implement (1-2 engineer days), portable across capable agentic harnesses (e.g., Claude Code), and performs strongly on the challenging task of CS resolution workflow generation, achieving 99.2% procedural equivalence to human-approved ground truth and 83.3% evidence sufficiency. Our analysis yields a concrete deployment insight: a skill-based agentic RAG implementation over a corpus augmented with a lightweight navigation layer can substitute for complex RAG infrastructure, shifting the primary maintenance burden from model artifacts to text and data curation.

Constrained decoding is essential in generative retrieval, where document identifiers generated directly from a query must exactly match a predefined library of valid IDs. At scale, decoding is often constrained using a trie with beam search but most implementations run on CPU. Limited parallelism then makes trie traversal and candidate validation a serving bottleneck as beam width grows.

We present FlashTrie, which addresses this limitation by optimizing constrained beam search on GPUs. It introduces an integer-aware succinct trie layout that uses bit compression to reduce memory footprint while keeping the full index in GPU high-bandwidth memory reducing memory stalls, and a cooperative CUDA kernel that performs beam expansion, validation, and pruning entirely on-device without per-step host orchestration. GPU-native parallel primitives further replace CPU-style irregular lookup and heap maintenance, improving warp utilization and reducing divergence.

Together, these designs significantly reduce decoding latency and increase throughput while preserving retrieval quality. On a library of 800M keywords with beam widths up to 1000, FlashTrie reduces trie-search latency to under 2 ms mean latency, achieving up to 24× speedup over a highly optimized multi-threaded CPU baseline. These improvements enable FlashTrie to scale beam sizes by up to 5× in latency-critical applications such as sponsored search. In a large-scale online A/B experiment on a commercial search engine, it delivers a statistically significant +0.71% revenue lift, enabling real-time constrained decoding at a scale previously impractical within our production serving budget

RAG systems are increasingly used to summarize what large collections of documents say. A user asks "What do people think about X?" and receives an answer that reads as consensus. But standard top-k retrieval ranks documents by query similarity, not by how faithfully they represent the population, so minority views quietly disappear. Existing fixes fall short. Diversity re-rankers like MMR and DPP spread retrieved documents apart, but with no target distribution to aim for. Calibration methods based on KL or JS divergence do target one, yet treat opinion bins as unordered: confusing strong positive with strong negative costs no more than an adjacent-bin miss.

We introduce WARP, a family of post-retrieval algorithms that calibrate retrieved evidence to the population’s opinion distribution. WARP first recovers underrepresented opinions that cosine ranking may bury, then uses Wasserstein-1 distance to select documents whose sentiment-intensity distribution matches the population target, capturing the ordinal structure ignored by KL and JS divergence. We develop three variants for dense, sparse, and variable candidate pools, trading off calibration quality and speed. Across three review domains spanning 35K documents, 156 queries, and 26 entities, WARP’s domain-matched variants reduce distributional error by at least 43% with sub-second latency. These gains carry through to generation: a five-judge LLM panel prefers WARP-generated answers in up to 86% of decided comparisons at k=5.

Warehouse operations are governed by Standard Operating Procedures (SOPs) that encode complex, multi-system decision logic, which must be executed reliably under strict time constraints, yet LLM agents lack mechanisms to enforce procedural compliance and degrade under the context overload full SOP specifications introduce. We present Eluna, a production-deployed agentic system for reliable SOP execution. Eluna is a graph-guided, multi-agent framework that encodes SOPs as directed acyclic graphs with progressive disclosure and delegates independent tasks to parallel sub-agents, each with persistent code execution and live data access. To meet production latency and accuracy needs, we use asymmetric episodic distillation where a strong teacher is improved through episodic error memories, then a smaller student is fine-tuned on the corrected trajectories with memory stripped, internalizing corrections without inference-time overhead. On a 13-task benchmark and two production applications, our fine-tuned models match or exceed their teacher, beat all larger off-the-shelf baselines, and reach 94% expert agreement on the ticket processing application.

We propose a multi-task learning approach for multi-party dialogue intent recognition that leverages an auxiliary task that models turn-taking dynamics. Specifically, we introduce turn-transition entropy, a self-supervised target computed from the sequence of speaker transitions, which quantifies the predictability of interaction patterns. Experiments on multiple pre-trained models demonstrate that incorporating this auxiliary task improves intent recognition performance, outperforming existing approaches which ignore multi-party interaction dynamics. We find that our proposed continuous target can be learned as a single-task objective, suggesting that it is an actual signal carrying useful information.

Context-heavy agents place substantial pressure on the key–value (KV) cache: long prefixes are reused across many short turns, while concurrency determines whether the serving system can keep GPUs utilized. We study 4-bit KV-cache compression for this setting, using TurboQuant-style rotation and codebook quantization as a quality anchor and vLLM FP8 KV caching as the deployment anchor. We report three contributions. First, we frame 4-bit KV caching around multi-round agent workloads where task quality, cache residency, and serving throughput must be measured jointly. Second, we describe the practical design choices needed to make the 4-bit path robust, including asymmetric K/V treatment, Walsh–Hadamard rotation, QJL removal, and block-scale variants. Third, we present serving optimizations on AMD GPUs, including optimized decode-attention kernels and UltraQuant, an FP4 approximation path that uses FP8 queries, FP4 KV tensors, UE8M0 group scales, and native scaled-MFMA support on CDNA4. On an adaptive-SLO replay of production Claude Code traces, UltraQuant sustains 2.71× (MiniMax-M2.5) and 4.38× (Qwen3-235B) the qualified-request throughput of the BF16 baseline, matching or exceeding hardware FP8 KV while using half the KV bytes. UltraQuant delivers its largest gains in long-context, high-concurrency, memory-constrained serving regimes.

Human-written references are typically treated as the upper-bound gold standard for fine-tuning and evaluation, but writing them is an inefficient use of subject-matter expert (SME) time. To best use limited SME attention, we apply psychometric standard-setting to discover acceptance criteria for GenAI applications. These acceptance boundaries are themselves useful for calibrating an LLM-as-a-judge, model tuning, and defensible production monitoring. We report a case study on summarization across two domains at a large US financial institution. While summarization is a classic GenAI use case, what counts as acceptable differs significantly across domains. 25 SMEs across three panels produced 2,500+ judgments and yielded calibrated thresholds in an estimated 10x less SME time than the standard playbook of writing golden reference (goldens) directly. In both domains, LLM-generated goldens are preferred over SME-written ones ~80% of the time. Limited SME attention is best used to judge candidate goldens rather than write them.

Enterprise analytics agents solve long-horizon tool-use problems over distributed business data, requiring retrieval, reasoning, API calls, code execution, and adaptation to intermediate observations. Supervised fine-tuning (SFT) calibrates tool syntax and teacher-supported behavior, whereas reinforcement learning (RL) can explore reward-supported behaviors beyond demonstrations; applied uniformly, however, RL can perturb already-calibrated skills. We study how to balance SFT and RL under production-mirroring beta APIs. We observe that, in our controlled experiment, checkpoint trajectories retrospectively separated into three regimes: Imitation, where SFT captured reliable teacher behavior; Lift, where both stages helped; and Discovery, where useful reward-observable behavior lay outside reliable teacher support. We leverage this prospectively, using teacher support and reward-observable headroom to route features to SFT only, SFT→RL, increased RL allocation, or further environment development. Across 18 subsequent feature-specific experiments, the diagnostic predicted 15/18 observed trajectories. On GPT-OSS 120B, targeted SFT→RL produced positive point estimates on 7/8 advertiser skills relative to a frontier Control; five positive gains had paired 95% confidence intervals excluding zero, while one skill had a confidence-supported regression. The largest gain was non-disclosure (+11.27 points; 95% CI [+9.72, +12.82]). A separate SME audit surfaced that targeted RL reduces standard leakage from 11.8% to 2.9% and adversarial leakage from 22.9% to 6.8% relative to SFT while preserving actionability (86.2% to 85.7%). In a matched uniform-versus-targeted comparison with shared rewards and optimization, targeted RL improved the seven-skill mean delta from +1.62 to +3.57 while using 43% less incremental RL compute.

Enterprise IT support knowledge graphs capture rich relationships among cases, users, devices, symptoms, taxonomic categories, root causes, and historical resolutions. Yet querying them in Gremlin requires knowledge of graph schemas, traversal semantics, edge directionality, and property-graph-specific constraints, making them difficult for non-expert operators to use. We introduce SEGRA, an experience-guided agent for enterprise text-to-Gremlin question answering. SEGRA integrates intent routing, schema- and taxonomy-grounded query generation, multi-shot decomposition, execution-aware verification, and a curriculum-bootstrapped skill library that reuses verified query patterns. On an enterprise IT support benchmark, SEGRA achieves a 7.0× higher mean judge score than backbone-only chain-of-thought prompting. Its skill library further reduces LLM calls by 20% and dollar cost by 18% relative to SEGRA without skills, while preserving answer quality. These results show that schema-grounded agent design and reusable execution experience improve both accuracy and efficiency for enterprise graph QA.

Recent advances in large Vision-Language Models (VLMs) have led to impressive performance on chart understanding tasks. However, existing evaluations are largely limited to clean, canonical charts and do not adequately measure robustness under realistic conditions. To address this gap, we introduce ROBUSTCHART360, a comprehensive benchmark for evaluating chart understanding robustness under realistic real-world variations. ROBUSTCHART360 spans six robustness dimensions covering textual and cross-modal interference, rendering and display degradation, accessibility-oriented transformations, and visual style variation. Using this benchmark, we conduct a large-scale evaluation of 14 proprietary and open-source VLMs across 2,505 chart instances and multiple robustness conditions designed to simulate practical deployment scenarios. Our analysis on 35,070 model responses reveals that current VLMs are highly sensitive to seemingly minor variations such as low-resolution rendering, and mobile layouts, with performance dropping by as much as 33.9% under certain conditions, such as modality gap. We further find that models often over-rely on textual priors when visual and textual signals conflict, indicating weak visual grounding despite strong performance on standard benchmarks. Our findings highlight a substantial gap between benchmark accuracy and real-world robustness, motivating the development of more reliable and visually grounded multimodal systems for chart understanding. We will release our code and dataset at <redacted>.

Large Language Models (LLMs) are increasingly used to annotate structured product data in e-commerce, but early deployment often begins as a cold-start problem: only limited pre-launch labels are available, the value of expensive reasoning is unknown, and human review is needed before the system can be trusted at scale. This challenge is especially common in rule-based annotation workflows, where each item must satisfy multiple business rules and both model errors and ambiguous rule boundaries affect final decisions. We introduce the Differential Reasoning Router (DRR), a cost-aware framework for cold-start LLM annotation that jointly optimizes model selection and human escalation. Rather than treating a reasoning model as a default fallback, DRR estimates separate success probabilities for a direct model and a reasoning model at both the sample and business-rule levels, enabling adaptive routing: easy cases are handled directly, reasoning is reserved for cases where it is expected to improve the decision, and likely double-failure or rule-disagreement cases are escalated to human annotators. The resulting labels provide targeted ground truth for prompt engineering, supervised fine-tuning, calibration, and rule refinement, enabling a gradual shift from human-heavy cold-start annotation toward high-confidence automated routing. In a production e-commerce workflow, DRR reaches accuracy parity with the strongest confidence-based router while achieving more than 60% reasoning-token cost savings.

We propose RCEM, a Robust Conversational search EMbedder that augments text embedders with LLM query reformulation capabilities while approximately preserving their original generalization performance. Unlike prior conversational dense retrieval approaches that learn direct conversation-to-passage matching, RCEM aligns conversations, prepended by special token, to LLM-rewritten queries, while attempting to preserve the original embedding space. The approximately preserved embedding space automatically maps the rewritten queries to the relevant passages. As a result, RCEM (1) reduces overfitting by aligning conversations to shorter rewritten queries rather than full passages, (2) eliminates the need for conversation-to-passage relevance labels for training, and (3) approximately preserves the original embedding space, enabling conversational retrieval over indexes built by the base embedder without re-indexing. Extensive experiments show that RCEM consistently outperforms prior approaches, improving up to 30% retrieval accuracy under distributional shift.

Supervised fine-tuning on teacher-generated trajectories is the standard first stage for distilling tool-calling capabilities into deployable models. Post-training pipelines that drive shipped tool-calling agents re-run this stage on a daily or weekly cadence, paying the frontier teacher cost each cycle, yet the mechanism is generate-and-filter (keep the teacher’s passing trajectories, discard the rest) and each cycle leaves behind the same hard scenarios because failures supply no signal. On τ2-bench, 57% of teacher trials fail, two-thirds of them near-misses (most tool calls correct, undone by one decisive error).

We introduce PROOF-Gen (Per-scenario Reflective Optimization to Overcome Failed Generation), which recovers golden trajectories from these failures via per-scenario prompt optimization. For each failed task, a reflector analyzes the execution trace and evaluation feedback, then writes corrective guidance that steers the teacher to a passing trajectory. The guidance is stripped before training, so the student learns from clean demonstrations with no task-specific scaffold.

On τ2-bench, per-scenario optimization recovers 93% of failed scenarios. Fine-tuned on the combined data, Qwen3-4B-Instruct-2507 improves from Passˆ1=0.132 to 0.529 and Gemma 4 E4B-it gains +7.2pp on BFCLv4 multi-turn. In a deployed pipeline, the method lifts trajectory quality by +6.3pp goal completion and transfers to a deployed on-device model (+1.5pp goal completion; +1.7 to +5.0pp across response-quality metrics), with positive transfer in every locale (non-English average +1.48pp).

Peer review plays a central role in scholarly publishing, yet verifying whether reviewer claims are supported by manuscript evidence remains a largely manual and time-consuming process. We present Peerify, a pipeline for manuscript-grounded verification of peer-review claims. Given a manuscript and a review comment, Peerify pipeline decomposes reviews into atomic claims, retrieves relevant manuscript evidence, and determines whether each claim is supported by the paper. To support the development and evaluation of the pipeline, we construct a benchmark of 800 claims derived from authentic peer-review interactions collected from NeurIPS 2024 and ICLR 2024, including a 300-claim hand-labeled subset used to audit the automated supervision. We evaluate state-of-the-art language models and retrieval strategies within Peerify pipeline, together with entailment baselines. Our results demonstrate the importance of retrieval-centered verification and claim decomposition, while highlighting the challenges posed by ambiguous and interpretive reviewer claims. Automated labels agree with human consensus on 90.3% of audited claims (𝜅 = 0.87), while off-the-shelf entailment models stay below 0.24 macro-F1.

On-device assistants require function-calling models that map natural language to local system actions, but existing resources emphasize web APIs or narrow mobile-action catalogs. We extend FunctionGemma 270M-it to practical Android workflows by introducing MOBILEACTIONSEXTENDED, a synthetic, schema-validated dataset of ∼9,500 conversations covering fifteen device-control categories, including messaging, phone calls, camera/screenshot, brightness control, device-status queries, flashlight control, and application management. We fine-tune the 270Mmodel with TRL supervised fine-tuning under completion-only loss, producing an extended specialist and a combined model trained jointly with Google’s MOBILEACTIONSGOOGLE. On MOBILEACTIONSEXTENDED, end-to-end accuracy improves from 29.3% for the base model and 17.2% for Google’s Mobile-Actions variant to 76.5%. The combined model retains 76.5% on MOBILEACTIONSEXTENDED and reaches 82.3% on MOBILEACTIONS-GOOGLE, down from the 90.3% of Google’s Mobile-Actions specialist, representing an 8.0-percentage-point trade-off in return for doubling category coverage. We release the dataset, fine-tuned models, reproducible training/evaluation pipeline, and an Android demo, highlighting compact local function calling as a practical path towards low-latency and privacy-preserving mobile assistants.

Production Retrieval-Augmented Generation (RAG) pays retrieval latency and serving cost on every query – even when the model already knows the answer. We present CONFRAG, a confidence-guided strategy that invokes RAG only when a large language model (LLM) is unlikely to be correct, cutting external retrievals by 5-39% and tail latency by up to 800ms (P90) in a real RAG system while matching the answer quality of always-on RAG. CONFRAG is enabled by CONFQA, a lightweight fine-tuning strategy (one epoch over 3K knowledge-graph facts) that teaches an LLM to answer only when confident and otherwise say "I am unsure", cutting hallucination rates from 20-40% to 5% or below across multiple factuality benchmarks. Two choices make it work in practice: a dampening prompt that discourages overconfident hallucinations, and training on atomic facts that generalizes confidence calibration to unseen domains and question types. At serving time, LLM-QA and RAG run in parallel with early-stopping, so confident answers return immediately.

LLM agents increasingly rely on external skills retrieved at runtime, making skill selection from large repositories a critical challenge. We present a production skill router over 34,396 skills and a large-scale study of skill retrieval using limited real supervision and synthetic data. We found that the synthetic-data fine-tuning improves in-distribution retrieval but it causes catastrophic forgetting on real and out-of-distribution (OOD) data. We evaluate several forgetting mitigation fine-tuning approaches inspired by continual learning, including embedding-anchor regularization, Learning without Forgetting (LwF), Elastic Weight Consolidation (EWC), and L2-initialization. The results show that these approaches not only retain the performance on OOD skills retrieval but also improve the retrieval on synthetic in-distribution skills by 13.98% for 0.6B Qwen retriever and reranker. Our results provide a practical benchmark and a robust fine-tuning recipe for scarce, multi-positive supervision.

Entity recognition in Automatic Speech Recognition (ASR) is challenging for rare and domain-specific terms. In domains such as finance, medicine, and air-traffic control, these errors are costly. If an entity is entirely absent from the ASR output, text-only post-correction has little signal to recover it. To address this, we introduce RECOVER, an agentic post-ASR entity-correction framework. It leverages multiple hypotheses as evidence from ASR, retrieves relevant entities, proposes Large Language Model (LLM) correction under constraints, filters them through deterministic guardrails and uses an LLM critic that judges each surviving edit and can trigger a targeted feedback round to recover an entity the first pass missed. The hypotheses are used by one of several strategies, namely 1-Best, Entity-Aware Select, Recognizer Output Voting Error Reduction (ROVER) Ensemble, LLM-Select and Adaptive Routing. Evaluated across five diverse datasets over a Whisper-large-v3 baseline, it achieves up to 49% relative reduction in entity-phrase word error rate (E-WER) and increases entity recall by up to 14 percentage points while preserving the overall word error rate (WER), with LLM-Select the strongest strategy.

Natural language processing (NLP) has been widely used in quantitative finance, but traditional methods often struggle to capture rich narratives in textual data, leaving potentially informative signals under-explored. Large language models (LLMs) offer a promising alternative as they are able to extract nuanced semantics. In this paper, we ask whether semantic signals extracted by LLMs from corporate disclosures predict abnormal returns (alpha) beyond broad market movements and common risk factors. We introduce a simple framework, *LLM as extractor, embedding as ruler*, which extracts context-aware, metric-focused textual spans and quantifies semantic changes across consecutive disclosure periods using embedding similarity. This allows us to measure the degree of metric shifting - how much firms move away from previously emphasized metrics, referred to as moving targets. In portfolio and cross-sectional regression tests against a recent traditional baseline, our method yields a moving-targets signal that delivers significant risk-adjusted alpha. Qualitative analysis suggests that these gains stem from preserving contextual qualifiers and filtering out non-metric terms that traditional approaches often miss.

Cross-lingual information retrieval (CLIR) is increasingly important in multi-national industries, where critical technical evidence may exist in a different language than the query. However, existing benchmarks do not adequately capture domain-specific cross-lingual retrieval or the retrieval-depth and recoverability failures that aggregate recall hides.

In this work, we benchmark CLIR in the chemical domain, with a focus on patent data. We construct a multilingual dataset from Google Patents and the European Patent Office (EPO) data, spanning five languages (covering major Eastern and Western languages) and reflecting the diversity and complexity of real-world industrial documentation.

Using this dataset, we systematically evaluate eight state-of-the-art embedding models for cross-lingual retrieval. Our results show a substantial performance gap between monolingual and cross-lingual settings: for the best-performing model, Recall@10 drops from 0.72 to 0.53 in cross-lingual setting. Retrieval depth also degrades significantly, with relevant documents ranked lower across languages in cross-lingual scenarios. Furthermore, some multilingual embedding models that perform strongly in monolingual settings exhibit sharp declines when queries and documents are in different languages, providing practical insights for model selection in cross-lingual use cases.

These findings highlight critical limitations of current approaches and emphasize the need for more robust cross-lingual retrieval methods in domain-specific settings. Our benchmark provides actionable insights for model selection and establishes a controlled diagnostic evaluation framework for CLIR over industrial technical text. Data and code are publicly available at https://github.com/MohammadKhodadad/Multi-Lingual-QAC.

Fixed benchmarks are costly to renew and cannot adapt their questions to model-specific failures. We ask whether LLMs can instead discover one another’s weaknesses and turn those observations into an evaluation process. To study this question, we introduce LivingArena, an automated peer-probing framework in which models take turns testing one another. Using the interaction history, each questioner identifies potential weaknesses of its opponent and constructs targeted, verifiable questions to probe them. A 3,600-round tournament of ten models reveals a clear role asymmetry: strong answerers are not always reliable questioners, because they may generate internally inconsistent tests or fail to verify their own reference answers. After a questioner exposes an answerer’s failure, it is more likely to pursue the same capability domain, while the answerer’s weakness recurs on independently generated questions, including questions written by different models. These findings show that peer probing can reveal persistent model-specific weaknesses while separately evaluating answering and reliable test construction. By automating this process and allowing test difficulty to evolve with model capabilities, LivingArena provides a “living” benchmark for model development, red-teaming, and capability-aware multi-agent coordination. We publicly release our code.[<https://github.com/galaxyChen/LivingArena>]

Long-running language-model systems accumulate interaction history that outgrows the context window, so they must continually evict. When an eviction policy drops a task-critical detail, for example an access token issued at login or a path the next call needs, the action fails. We present LRE (Learned Relevance Eviction), a kilobyte-scale, CPU-only, language-model-free scorer that learns which units of history are task-critical and keeps them by verbatim extraction. Under a matched-budget comparison, in our experiment, no baseline dominates LRE on the accuracy-cost plane. On agents, LRE recovers 93% of the aggregate accuracy of keeping the entire history (41.1 vs. 44.0) and exceeds it by 27% on the simplest tasks, while requiring zero compressor calls and cutting the worst-case peak prompt by 52%. A controlled study trace shows LRE completes tasks where the others loop, finishing one such task in 37% fewer calls than keeping everything and solving 14 tasks where no other run policy does. On conversational memory, LRE outranks dense and token-pruning encoders at zero neural cost while being 295–1569× smaller in size. In downstream evaluation, LRE gives the best budgeted answer quality on LoCoMo reading 68% fewer tokens. Its supervision can also be annotation-free: training only on the system’s own behavior recovers 95% of the supervised scorer’s effectiveness. We argue that, because memory eviction in LLM agents is a fidelity problem, it requires a deployable proactive policy where the future query is unavailable and exact state is decisive, and that cheap learned relevance can be sufficient.

Fidelity metrics such as per-token KL divergence (KLD) against a high-precision reference are often used as low-cost proxies for benchmark quality. We test this practice on 28 Qwen3.6-35B-A3B and 41 Devstral-Small-2-24B quantizations across downstream benchmarks. KLD strongly correlates with benchmark score over the full cohorts (𝜌=-0.72 on Qwen and 𝜌=-0.86 on Devstral, both with p<0.001). However, among quantizations retaining at least 99% of measured BF16 composite quality, KLD provides no dependable ranking signal (𝜌=+0.11, p=0.71, n=14 on Qwen; 𝜌=+0.02, p=0.93, n=16 on Devstral). This collapse persists across measurement variants spanning KLD aggregation, perplexity formulations, top-1 agreement, measurement corpora, and context lengths. At the per-prompt level, KLD provides only weak failure-prediction power and fails as a cross-model router across four benchmarks, selecting the better response on only 40.0-49.1% of disagreement prompts and reducing score relative to the better static model on all four. We trace the collapse to a structural decomposition: within the KLD-defined silent zone, KLD primarily measures the volume of disagreement with the reference (composite 𝜌=+0.94, p<0.001 on Qwen; 𝜌=+0.55, p=0.03 on Devstral), while its relationship to the direction of those disagreements is weak and task-conditional.

Enterprise cloud agents increasingly operate over thousands of APIs and SDK operations. Tool use therefore becomes a large-scale classification and grounding problem over API operations, not a simple function-calling task. Earlier production-derived architectures either expose broad operation catalogs to the model or hide selection inside planner agents, causing excessive context growth, repeated discovery calls, and preventable invocation errors. We propose the Operation-Visible Agent Architecture, which keeps operation search, schema inspection, validation feedback, and API invocation visible to a centralized ReAct loop through a deterministic Operation-Resolver MCP. We compare this architecture with lead-worker planning and ReAct-based execution variants to assess how agent orchestration and MCP tool-boundary design affect enterprise-agent performance. On internal cloud-management benchmark tasks spanning 280+ SDK clients and approximately 8K operations, the proposed architecture improves full-success rate from 68% to 77% over the production-derived baseline, while reducing total token usage by 54.1%, median latency by 43.3%, LLM calls by 37.8%, and tool errors by 52.4%. The main lesson is that, at enterprise API scale, tool-boundary design is not an implementation detail: parallel tool calling improves efficiency, but task success improves most when operation resolution remains visible to the main reasoning loop.

Prompt-level glossary injection can recover ASR-corrupted domain terms, but its effectiveness depends sharply on glossary scope. In enterprise meeting summarization, a black-box STT system passes only a 1-best transcript to the summarizer; its errors on loanwords, acronyms, and domain terms can replace the canonical forms needed downstream. Motivated by the keyword-input UX of a deployed Korean meeting summarization service, we evaluate training-free prompt-level injection of an annotation-derived meeting-scoped glossary proxy without transcript rewriting. On 30 public Korean meetings processed offline by operational STT, 27% of annotated terms are corrupted under normalized matching, and a no-glossary summarizer recovers only 1.3% of these corrupted terms. Meeting-scoped injection raises term recovery to 53.8%, with 89.2% of output keywords grounded in the transcript. The effect collapses under broader scopes: on the same 45 terms, meeting-level recovery is 66.7% versus 6.7% for a 2,669-term full-dictionary proxy. Coverage-controlled size sweeps show this collapse on Qwen3-8B, and three EXAONE-3.5 points reproduce it, indicating prompt-level over-biasing. A four-annotator human evaluation does not establish improved summary utility; we therefore treat domain-term recovery as an operational dimension, not a summary-preference proxy. We contribute a public-data evaluation protocol and deployment analysis of domain-term recovery under black-box STT constraints.

Table extraction suffers from frequent structural errors and semantic hallucinations. We propose PrismAlign, a multi-VLM framework aligning diverse visual perspectives to resolve ambiguity. It integrates priors of table logic to assess output plausibility, decoupling structural alignment from cell content alignment. A Bayesian decision strategy maximizes alignment accuracy by exploiting the correlation between extraction errors and computable rule violations. Evaluated on open-source and custom VLMs, PrismAlign significantly reduces hallucinations and achieves state-of-the-art performance on OmniDocBench 1.5, as well as on the table category of CC-OCR and PureDocBench.

Production LLM agents increasingly depend on real-time search, yet native search grounding bundles retrieval policy, provider choice, evidence injection, cost, latency, and generation behavior behind a single model-provider boundary. This coupling makes grounding hard to inspect, tune, reuse, or port, and can trigger Search-Induced Verbosity that breaks strict output contracts. We present Decoupled Search Grounding (DSG), a vendor-agnostic boundary that moves grounding outside the reasoning model through an MCP-compatible gateway, exposing provider routing, source-aware context rendering, configured fallback, retrieval-depth control, and exact plus semantic caching as first-class controls. Across five frontier models on SimpleQA, FreshQA, and HotpotQA, native search leads on recency-sensitive FreshQA, but DSG exposes a stronger frontier when control matters: on SimpleQA it nearly matches native accuracy (86.1% vs. 87.7%) at 91% lower search cost, preserves concise answer contracts, and reaches a 99.4% warm-cache hit rate with 68% lower latency. Deployed as a shared production grounding layer for large-scale agentic workloads with interchangeable models, DSG matches native-search accuracy on an e-commerce query-understanding (QIU) workload while cutting search cost by over 98%. Real-time grounding is best treated as an optimizable interface boundary, not a fixed model feature.

Evaluating enterprise agents on domain-specific benchmarks is critical, yet public benchmarks rarely evaluate whether agents can integrate business knowledge with analytical computation, and constructing such benchmarks manually is costly. We present DI-Bench, a pipeline for generating realistic benchmarks for data intelligence (DI), the practice of extracting insights from large volumes of enterprise data. To emulate realistic DI tasks that require both computation and knowledge retrieval, DI-Bench builds an artifact linkage graph over data tables, dimensions, metrics, and documents to form questions involving structured data and associated knowledge. Ground truth answers are derived via query execution, followed by LLM question generation and validation. Applied to two public datasets, the pipeline produces a 731-task benchmark covering knowledge retrieval, analytical computation, and rule-grounded reasoning. To show the discriminatory capability and difficulty of the benchmark, we evaluate four models, revealing a substantial finding: models achieve only 32% accuracy when doing computational tasks where retrieved business rules modify the computation.

Vision Language Model (VLM)-based OCR-free pipelines are increasingly adopted for industrial Key Information Extraction (KIE) systems due to their simpler end-to-end architecture and reduced dependence on potentially noisy OCR outputs. However, their reliability in grounding extracted values to precise bounding boxes remains unclear, raising questions about their suitability for grounding-critical use cases. In this paper, we present a systematic study of bounding box grounding capability in both general and document-specific VLMs. Our findings reveal a consistent extraction-grounding gap across model families. We revisit the OCR system as a grounding primitive and find that delegating the grounding step to Large Language Models (LLMs), rather than VLMs, in a zero-shot manner substantially improves region-level grounding accuracy. We introduce a fine-tuned 0.6B LLM that further improves the predicted box alignment, achieving high overlap with the target field for in-domain data while maintaining low serving latency. These results position OCR-anchored grounding as a practical deployment choice for industrial KIE systems that require both extraction and precise localization.

An advertising platform typically generates a diverse candidate set of text creatives for each ad and then employs personalized online ranking to estimate the top-performing one for final display. Conventionally, the diversity is treated as a by-product during the stochastic decoding process rather than a training objective, which results in suboptimal outputs due to the lack of explicit modeling for semantic coverage. In this paper we reformulate ad text generation as a set-level modeling problem, and propose a parallel generation framework that optimizes the diversity objective with set-level alignment. By explicitly decomposing an ad’s content structure into semantic-independent perspectives via a planning mechanism, our framework generates the entire set of text creatives in parallel based on the planned semantics, which also takes full advantage of parallelizability to enable high-throughput industrial deployment. It is optimized through a set-level alignment stage that encourages the overall semantic coverage and penalizes the redundancy among the candidates. Comprehensive experiments on benchmark dataset show that our framework achieves better set-level quality and diversity, as well as significantly improving generation throughput, demonstrating its superiority on both generation quality and inference speed.

Financial retrieval-augmented generation (RAG) systems typically rank evidence by textual relevance, but in financial markets evidence utility depends on event type, forecast horizon, and market context. We study news-triggered event-impact prediction as a point-in-time financial RAG problem. For each company-news anchor, the system retrieves financial news and SEC filing passages, appends a pre-decision market-context card, and predicts multi-horizon residual-return signals. Our method keeps the LLM reader frozen and adapts retrieval through an external Bayesian source memory updated from matured residual-return feedback. On a fixed 89-stock FNSPID universe with point-in-time EDGAR evidence, Frozen Reader with Source Memory improves held-out macro-F1 from 0.438 to 0.471 and downstream portfolio Sharpe from 0.52 to 0.84 relative to Frozen Reader with No Memory. Supervised LoRA gives modest gains under static retrieval, but after source-memory adaptation, the LoRA reader does not improve over the frozen reader. These results suggest that, for financial RAG systems, learning where to retrieve can be as important as learning how to read, offering a modular route to market-feedback adaptation.

Fine-tuned small language models (SLMs) are attractive defaults for large-scale structured at- tribute generation—producing values such as materials, colors, or dimensions from product descriptions—as they are inexpensive to serve. However, a fine-tuned SLM is not reliable ev- erywhere as its correctness varies sharply, so deploying it still requires deciding, per output, whether to keep the SLM’s value or escalate it to a more capable fallback. The standard pol- icy applies a single confidence threshold, but one threshold cannot separate scores on reli- able tasks from those on unreliable ones, so it reduces coverage at no gain in quality. We demonstrate that in a fixed deployment scope, where the target tasks recur and validation data is available, a lightweight scope-aware router using only SLM-side features (token confi- dence plus task-scope identity) selects risky predictions far better. Our experiments show that at a fixed precision target it accepts about 11 points more values than a global confidence threshold (family-level calibration helps only marginally). Ablation experiments confirm the gain comes from task-scope reliability priors rather than a better confidence transformation. The router is an 8-feature gradient-boosted tree that adds negligible inference cost, making a strong fine-tuned SLM the low-cost tier of a cascade that optimizes the quality–cost tradeoff in real production settings.

High-quality preference data collection remains a bottleneck for RLHF reward model training: collecting multiple independent human annotations per sample is costly and hard to scale across domains and languages, while fully automated AI feedback (RLAIF) risks quality degradation. We present a production-grade hybrid human-AI annotation framework that introduces automated model-as-annotator pre-annotations into a quality-control (QC)-ratified workflow, contributing model votes only when automation is sufficiently reliable and leaving final adjudication to human QC. The pipeline (1) verifies factual claims with a web-search-grounded fact checker under a cleanroom constraint that keeps proprietary tokens out of training data, (2) elicits structured judgments from an ensemble of open-weights LLM judges, and (3) applies a two-call swap-consistency gate to mitigate position bias, with a data flywheel surfacing disagreements between QC and model outputs to drive improvement. In production, the framework reduces initial-labeler hours by ∼37%; across 11,092 production comparisons it reaches 81.6% agreement with QC (𝜅=0.63) and 82% to 88% on the public JudgeLM benchmark, with no QC degradation measured under an independent-model proxy. The resulting data supported a production reward model and aligned LLM.

Long-term memory systems allow LLM agents to preserve information beyond a single context window, but most systems focus on storing and retrieving facts after extraction, leaving the write decision under-specified. What deserves memory can depend on the user’s current task, topic, activity, or interaction partner, while uniform extraction applies one notion of importance across these different situations. We formulate this challenge as preference-conditioned write control and introduce AdaMem, which uses adaptive natural-language Memory Policies to personalize what an agent writes to memory. Each policy represents the user’s memory preference for a particular interaction context, is updated from periodic feedback, and controls subsequent memory writing. We evaluate this loop in AdaMem-Bench, which assigns different memory preferences to six concurrent interaction personas across five ten-week stories. Across two extraction models and two feedback modes, AdaMem improves average QA accuracy over Mem0 from 80.0% to 84.35% while reducing persistent memory by 9.27%. Our analyses show that explicit feedback helps models learn better memory policies, but current models still struggle to translate those policies into reliably selective writing behavior. AdaMem thus demonstrates the promise of adaptive write control while exposing policy execution as a central limitation of current memory agents. Our code is publicly available.[<https://github.com/galaxyChen/AdaMem>]

Comparing and selecting task-oriented LLM agents increasingly relies on a low-cost offline evaluation gate: persona-driven LLM user-simulators converse with each candidate, an LLM-as-a-judge scores the transcripts, and the higher-scoring agent is promoted. We introduce GAUGE, a reusable offline protocol that measures whether this gate’s ranking matches a grounded verifiable reward across 25 agents from six providers on the 𝜏²-bench and SimulatorArena benchmarks, separating two kinds of evaluation validity that release practices conflate: ranking validity and construct validity. First, a satisfaction-success gap: satisfaction carries essentially no information about task success, as conversations rated satisfied by our blind panel are decorrelated from actual success, with 57.5% of them failing the customer’s task, a pattern consistent across five rater populations, both benchmarks, and every subjective dimension we rated. Second, while the gate’s ranking is robust across the broad capability span, it loses resolution among the near-equal strong agents: this decision-disagreement rate jumps from <1% on wide-reward pairs to 31% on close pairs. The gate is thus human-validated yet mis-anchored. As a remedy, we propose a calibrate-then-trust cadence in which a judge-free completion bit is a zero-cost tripwire for truncation regressions.

Local search systems must determine WHERE to search before retrieving results. We present OmniQUM, a fine-tuned 7B model for generative location disambiguation: given a query and geo-coordinates, it produces a disambiguated location target and bounding box decision in a single pass. Unlike NER + gazetteer pipelines, it jointly reasons over query intent and geographic context. We train across 14 locales and 7 languages with a failure-driven agentic pipeline generating targeted samples iteratively. We introduce expanding donut retrieval, sampling candidates from concentric, slightly overlapping rings to improve recall over naive bounding box retrieval. On 52K queries reflecting real traffic distribution, we achieve F1 0.94 on location disambiguation and 32.7% retrieval recall improvement, surfacing 86K additional relevant POIs. Deployed with vLLM and bf16 on dual GPUs, we achieve sub-150ms TM95 latency at 0.12× baseline cost, with 7–38% cache hit rates.

Product families in large product catalogs suffer from inconsistencies such as duplicates and unit mismatches that degrade customer experience. Detecting these requires reasoning over multiple error types across lengthy product listings, where LLM classification quality degrades due to long-context limitations. We address this by decomposing detection into focused sub-tasks that reduce context and isolate error types, improving F1 from 52% to 87%. For scalable deployment, we introduce Positional Task Conditioning (PTC), which distills this capability into a single smaller model by reinforcing task identity at structural prompt boundaries. PTC outperforms rationale-based distillation across five models and two architecture families, achieving within 1.79% F1 of the frontier at upto 98% lower cost. Our system is deployed across multiple countries processing 10+ million product families.

MagicSchool’s K–12 AI product suite is used by millions of teachers and serves millions of teacher and student messages each month. Our team monitors output along four priority dimensions—student safety, tone and instructional role, pedagogical value, and structural output quality—and tracks how often it fails on each. As our program matured, false positives came to dominate the evaluators’ flags, misdirecting scarce analyst attention away from failures that warrant product change. To address this, we deployed three enhancements—unanimous-fail panels of repeated judge runs, per-evaluator judge-model choices, and softened rubrics—backed by a pair of synthetic datasets: a benchmark that measures how often cases are flagged, and an egregious-failure set as a check that severe cases still fail. That set is human-reviewed, with 12 to 79 cases per evaluator, each designed to exhibit an unambiguous violation of the failure mode it targets. Across 21 deployed evaluators, final configurations reached a median benchmark activation rate of 0.04% (16 of 21 at or below 0.2%; 8 at 0.00%). The strategy cuts confirmed false-positive flags by 99% and raises per-flag precision from 0.6% to 49%, while holding egregious-failure capture at 100% and improving it on 8 of 21 evaluators.

As agentic LLM systems move from prototypes to deployment, evaluating them has become more important and more difficult. Evaluation goals are often left implicit. Without a clear account of what a system is expected to do, how it can fail, and which failures matter, metric choices become difficult to justify, interpret, or validate. We present Litmus, a zero-label system that designs evaluation and monitoring metrics for AI pipelines by eliciting evaluation intent from source code and targeted interrogation. Litmus first identifies what must be measured and why, then converts those answers into constraints for a justified, per-stage metric portfolio. We evaluate Litmus on three real, code-defined pipelines against AutoMetrics and three DynamicRubric baselines. Litmus achieves broader concern coverage on account grouping at matched size, spans more stages, produces a near-zero-redundancy portfolio, and, under a zero-label design-time constraint, ranks first in validity against per-row quality labels on all three pipelines, decisively on scientific QA (𝜌=0.72 vs. 0.47 for the strongest baseline on the stitched single-vector score), with overlapping intervals on the other two. Results support a shift from automatic metric implementation to automatic metric specification: before asking which metric to compute, evaluation systems should ask what must be measured and why.

Voice agents increasingly issue tool calls that disclose private information or modify external state. Correct intent recognition is not sufficient for authorization when identity, entity values, confirmation, or policy evidence is stale, missing, or contradictory. We introduce VoiceAgentGuard, a model-agnostic runtime gate that separates a planner’s proposed action from release. For every private-read or mutating call, it builds a Voice Action Contract from independently maintained identity state, entity revisions, confirmation evidence, policy support, schema validity, risk tier, and handoff requirements. A deterministic, side-effect-free validator either executes the call or returns a repair action: clarify, confirm, block, or handoff. SafeOutcome credits policy-correct completion, blocking, and escalation while penalizing unsupported execution. On a 480-task GPT-5.5 transfer, VoiceAgentGuard reduces unsafe release from 0.250 to 0.017 relative to an LLM guard with comparable pass@1. On 2,400 perturbed traces, it improves SafeOutcome from 0.417 to 0.733 and lowers unsafe release from 0.568 to 0.100. On 480 held-out adaptive attacks, unsafe release is 0.075 versus 0.340 for the LLM guard, with 1.55 ms median warm-path latency. Grounded, live-planner, and real-speech checks show that explicit contracts turn insufficient evidence into auditable repair rather than unsafe execution.

Agentic retrieval-augmented generation (RAG) systems in business-to-business settings may often receive free-form response feedback. Rather than generic feedback signals such as style, preference, or overall response quality, we focus on actionable factual corrections. We identify these instances and convert them into compact knowledge-base entries, which we call factual nuggets. We introduce Iterative Nugget Optimization (INO), an index-time optimization method that uses the production agentic RAG as a test harness: it creates an initial nugget, probes it with the triggering query and paraphrases, reflects over failed retrieval and answer traces, and revises the nugget until it is discoverable. We evaluate INO with two production B2B knowledge-assistance agents across multiple companies that use our system: a product support agent that answers questions over company-specific knowledge bases, and a support ticket agent that assists support engineers. INO consistently improves results over baselines in terms of discoverability and usage of factual corrections, in automated and human evaluations. A controlled experiment in production shows that the nuggets improve user satisfaction with responses.

For industrial content risk control, the real deployment constraint is not average accuracy but how much risk can be auto-handled under high precision and second-level latency. We present SIRF (Spec-Internalized Risk Foundation Model), which internalizes a platform’s complex policies, synthesized label-free via EntiGraph, MAGA rewriting, and account-level CoT, into the weights via continued pretraining, so rules are applied at high precision under an ultra-low-latency, classification-only deployment. A controlled same-source comparison (Qwen3-8B-SFT vs. SIRF-8B-SFT, identical policy injection and classification-only form, differing only in policy-grounded CPT) attributes the gain to internalization: SIRF-8B-SFT reaches 71.3 Black Recall@P95, +15 pp over the baseline, using only about 70M CPT tokens without harming general ability, and among included, logprob-available models under this interface it matches or exceeds far larger systems. SIRF is deployed as a tree-model adjudication layer (+20% more mis-penalized samples recovered) and transfers to a freezing scenario at low cost (~70% relative mis-penalization reduction).

Seller-authored product titles often contradict their own bullet points (a title reading “128GB” against a bullet claiming “512GB”), which misleads shoppers and drives returns. At catalog scale, detection is precision-binding: even a small false-positive rate flags a huge absolute volume of good listings. We deploy a two-stage cascade on one national storefront of a large e-commerce marketplace: a compact 4B detector flags candidates and a strong reasoning model (DeepSeek R1) verifies them, reaching >99% precision while scoring all 36M listings. Verification is the expensive stage, so our goal is to fold it into the cheap model and run on the 4B detector alone; until then, every gain in the detector’s recall catches more defects per verifier call and lifts coverage. We improve that first stage with Contrastive Token Weighting (CTW), a per-token credit-assignment method for GRPO. GRPO spreads one sequence-level advantage uniformly across a completion, even though only a few pivotal tokens decide correctness; CTW instead reads credit from a signal the reward already computes, the split of each sampled group into correct and incorrect completions, and upweights positions where a frozen scorer tells the two apart. The scorer can be a separate stronger teacher (CTW-T) or, when none is available, a frozen copy of the student itself (CTW-S, teacher-free); the teacher-free variant is the one we deploy, and it lifts the detector to 85.9 F1 (+2.2 over SFT, with the gain concentrated in recall, +3.4). A controlled math study shows the reward-induced partition, not the reweighting itself, is what drives the gain.

Awakening dormant users, who remain engaged but exhibit low conversion, is a key driver for incremental GMV growth in e-commerce platforms. Existing methods often underperform since they typically rely on single-step estimation of an item’s intrinsic value (e.g., immediate click probability). This overlooks the instrumental effect of items, where specific interactions act as triggers to shape latent intent and drive subsequent decisions along a conversion trajectory. To bridge this gap, we propose RoleGen, a framework that synergizes a Conversion Trajectory Reasoner with a Generative Collaborative Executor. Specifically, the LLM-based Reasoner explicitly models the context-dependent Functional Role of items to reconstruct intent evolution. It further uses counterfactual role exploration to mitigate interest collapse and generate diverse conversion-oriented candidates. These reasoned candidate items are then injected into the Executor, which is optimized via a feedback-driven co-training mechanism to ensure grounded execution. Extensive offline experiments and online A/B testing on a large-scale e-commerce platform show a 5.82% gain in Recall@1 and a 6.71% increase in online order volume, confirming RoleGen’s effectiveness in activating dormant users.

In enterprise applications, document information extraction (IE) does not operate in isolation: extracted information must match the systems of record (SoR) for the process to work. While reliable IE demands entity extraction, SoR grounding, and iterative refinement, existing methods rely on single-pass inference. To address this gap, we propose a SoR-grounded agentic pipeline and evaluate four agentic paradigms (Agentic Workflow, ReAct, Plan-Execute-Critic, and Dynamic Agent Collaboration) on real-world enterprise data. We focus on the goods receipt and invoice settlement processes as representative use cases, where delivery notes and invoices must be matched and verified against purchase order SoR. Our results show that agentic approaches generally match or exceed non-agentic baselines, with simpler paradigms matching or outperforming complex alternatives. Paired with systematic error analysis, this work provides actionable guidance for deploying agentic document AI at enterprise scale.

Large language model (LLM) agents are increasingly deployed over Model Context Protocol (MCP) servers, yet the benchmarks used to evaluate them score the final answer or a fixed “ground-truth” list of tools, both of which are fragile once the underlying data is live and stateful. We present DynamicMCPBench, a reusable framework rather than a fixed dataset. A practitioner can run it on their own MCP servers to test models on their own tasks, or let it collect servers automatically to measure a model’s general ability to solve agentic tasks. Given the servers and any set of models, it generates realistic goals, pursues each one live to record a successful trajectory, distills that trajectory into path-agnostic effect checkpoints, and scores an agent on whether it reproduces those effects, never on the final answer. To show what the framework reveals, we run it at scale: 24 models over 121 servers and 750 tasks spread evenly over 15 task categories (50 each), where each category targets a distinct tool-use challenge of the generated questions. Each task is scored by pass^3, it counts as solved only if all three independent attempts succeed. Even the strongest agents solve only about half of the tasks, 31% of tasks are solved by no model at all, and accuracy collapses as the required tool chain grows longer (from 39% on the shortest chains to 13% on the longest). A human validation study confirms the automatic scoring is reliable (chance-corrected agreement of 0.76). DynamicMCPBench thus turns benchmark construction into something practitioners can rerun on their own servers and models, while exposing a consistent inability of current agents to handle long, multi-step agentic tasks.

Agentic LLM frameworks that rely on prompted orchestration, where the model itself determines workflow transitions, often suffer from hallucinated routing, infinite loops, and non-reproducible execution. We introduce GraphBit, an engine-orchestrated framework that defines workflows explicitly and deterministically as a directed acyclic graph (DAG). Unlike prompted orchestration, agents in GraphBit operate as typed functions, while a Rust-based engine governs routing, state transitions, and tool invocation, ensuring reproducibility and auditability. The engine supports parallel branch execution, conditional control flow over structured state predicates, and configurable error recovery. A three-tier memory architecture (ephemeral scratch space, structured state, and external connectors) isolates context across stages, preventing cascading context bloat that degrades reasoning in long-running pipelines. Across GAIA benchmark tasks spanning zero-tool, document-augmented, and web-enabled workflows, GraphBit shows excellent performance achieving the highest accuracy, zero orchestration errors, the lowest latency, and the highest throughput. Ablation studies demonstrate that each memory tier contributes measurably to performance with similar configuration due to architectural changes, with deterministic execution providing the greatest gains on tool-intensive tasks representative of real-world deployments.

Code and data: https://github.com/InfinitiBit/graphbit

Benchmark infrastructure for personally identifiable information (PII) detection remains limited: existing corpora cover few entity types, use ad hoc generation conditions, and do not show which surface conditions cause detector failures. We present REDACT, a systematically controlled multilingual PII benchmark with 13,427 records, 324,078 entity annotations, 51 entity types, 4,127 surface-form patterns, and 25 languages across 9 scripts. A strength-2 covering-array sampler controls nine generation axes: domain, format, difficulty, length, density, code-switching, language, adjacency, and co-occurrence. Three entity-level metadata fields (disclosure status, disclosure form, and a GDPR-aligned sensitivity tier) enable stratified evaluation beyond aggregate or per-type F1. From the full benchmark, we evaluate five detectors (Presidio, GLiNER, the OpenAI Privacy Filter, GPT-4.1, and Claude Sonnet 4.6) on a locked, language-stratified sample of 1,000 records. Aggregate F1 masks an architecture-dependent failure structure: the rule-based detector performs poorly on the policy-sensitive data, including HIGH-sensitivity entity types (recall 0.07) and non-verbatim disclosure forms, while the LLM detectors remain more robust, with the HIGH tier as their strongest sensitivity slice. A three-model reference-free LLM-as-judge assessment corroborates that sensitivity-tier assignment is the task’s hardest axis. We release the benchmark, schema, prompts, and stratified evaluation harness.

During disasters, social media becomes a critical channel for identifying urgent needs and mobilizing public and institutional response efforts. Prior crisis-informatics research has mainly focused on detecting or extracting actionable content from disaster-related posts. However, real-world humanitarian response requires structured and operationally complete records, while social media posts are often short, informal, and underspecified.

In this work, we present CrisisLink, an LLM-based coordination system that transforms noisy crisis communications into structured crisis records. Unlike prior one-shot extraction systems, CrisisLink dynamically identifies missing operational fields and initiates clarification dialogues with users to repair incomplete records before downstream coordination. The system further grounds resources against a humanitarian taxonomy and supports partial, multi-item matching.

Built on the CrisisNLP corpus spanning multiple disaster types, the system is evaluated through human-audited assessments, simulated dispatch workflows, and a pilot evaluation in which trained workers role-play poster replies through a live web-based response interface. Experimental results show that a single clarification round raises end-to-end Offer–Request matching accuracy from 74.0% to 88.3% (a ceiling estimate under full response)while reducing the manual triage burden on responders, demonstrating the potential of interactive LLM systems for real-world humanitarian coordination.

LLM agents deployed for software engineering fail expensively: they act confidently wrong, and bad actions are recognized only after costly execution and retry. We present Speculative Uncertainty (SU), a method that recovers a predictive failure signal for a black-box agent from its output tokens alone, with no access to logits, weights, activations, or repeated sampling. Inverting speculative decoding, a small open-weight draft model scores the agent’s already-generated trajectory in a single forward pass. From these speculative cross-likelihoods we extract phase-aware features by separating the reasoning and action spans, and calibrate them against a verifiable objective. SU produces a failure-likelihood score that any downstream policy, such as routing, human intervention, or extra test-time compute, can consume directly. To show the signal is actionable, we instantiate one such policy, a pre-execution veto gate, on software-engineering agents Qwen3-Coder-480B and closed-source Claude 3.5 Sonnet, cutting execution error rate by 6-8 percentage points and token cost by 14-19% in deployment, transferring to out-of-distribution benchmarks without retraining, and generalizing across agent models.

Large Language Model (LLM) based conversational systems generate vast interaction logs containing critical diagnostic signals such as failure modes and safety violations. However, the cross-turn dependencies, implicit state, and topic drift inherent to these conversations defeat dense retrieval, whose lone embedding cannot concurrently encode discrete structured constraints (e.g., refusal type) and open-ended semantic content. Our central claim is conceptual: conversational retrieval must span two complementary information planes, structured metadata and semantic content, and these are best reconciled by a unified hybrid representation that jointly supports filtering over structured metadata fields and vector similarity search over unbounded open-ended fields. We instantiate this as the Extract-Annotate-Query (EAQ) framework, which derives a metadata taxonomy, applies LLM-driven annotation, and compiles natural-language queries into hybrid graph operations. On the CDR-Benchmark, EAQ lifts NDCG@10 by 30.8% over the strongest dense baseline, with consistent gains across five analytical categories and two external datasets.

Can an LLM read a 30+ page Standard Operating Procedure (SOP) and produce a team of agents that reliably execute it? Industrial SOPs bury workflow logic in prose, leave inter-step dependencies implicit, and scatter referenced entities across sections, forcing planners to parse, sequence, and tool-map within one context window. We argue that the missing primitive is procedural chunking: decomposing a document into self-contained executable units. Unlike RAG-style chunking, each chunk must be independently executable and chunks must jointly form a valid DAG encoding data and process dependencies, turning a prose SOP into a compact orchestrable workflow graph. We present ProChunker, a three-phase self-optimizing LLM pipeline (Boundary Detection, Metadata Extraction, Dependency Construction) with Stochastic Best-of-N prompt exploration guided by five custom metrics (Boundary Clarity, Entity Reference Coverage, Agent Self-Containedness Agreement, Step Coverage, Graph Validity); we then fine-tune Qwen3-1.7B on the pipeline’s own phase outputs, cutting latency by ∼27 min. Across three corpora (500 BioProBench protocols, six Industrial SOPs, 14 SOP-Bench documents), ProChunker improves the five-metric composite over the strongest baseline by +0.036 to +0.111 across corpora. Our study shows ProChunker increases agent construction success rate by +17% and resolves >90% of procedural ambiguity.

Applying LLMs to query-document relevance estimation in industrial document retrieval faces two key challenges: the complexity and heterogeneity of Web content, and stringent latency and resource constraints. To address them, we propose an automated framework with two components: **R**eflection-**A**ware **C**ollaborative **A**gent (**RACA**) and **P**rogressive **O**ptimization **D**istillation **S**trategy (**PODS**). RACA uses collaborative reasoning and reflection to produce relevance labels and structured rationales. PODS stratifies training data by difficulty and combines supervised fine-tuning with Reasoning-Informed GRPO. Rationale supervision improves the student’s first-token relevance decision, while deployment stops after that token and avoids explicit rationale generation. Together, these mechanisms enable reasoning-informed first-token prediction that preserves the benefits of structured rationale supervision under stringent latency and resource constraints. Extensive offline experiments and a 10-day online A/B test in WeChat Search demonstrate the resulting accuracy–efficiency trade-off.

Enterprise forms are layout-specific observations of real-world entities whose semantics span fields, recurring sections and related documents. Developing enterprise document-understanding systems requires large-scale supervision, but obtaining authentic forms is constrained by privacy concerns, annotation costs, and organization-specific formats. We present , a semantic-first framework that uses a lightweight, task-specific ontology to generate enterprise form packets. The framework separates domain semantics from layout through ontology maps, hierarchical schemas, dependency-aware entity sampling, deterministic validation, and template bindings. It produces rendered forms with field boxes, ontology-keyed extraction targets, grounded question-answer pairs, and validation metadata. We instantiate the framework on three digitally rendered surrogate insurance form types, producing 706 forms. Finetuned vision-language models improve ontology-keyed extraction F₁ by 4.99 points and grounded-QA ROUGE-L by 44.1 points over their zero-shot baselines. Ontology guidance further improves held-out-form extraction F₁ by 36.05 points overall and by 42.67 points on concepts absent from training, while increasing exact cross-form reasoning accuracy by 34.17 points. These results demonstrate the utility of the generated supervision signals for downstream tasks

Federated learning (FL) with differential privacy (DP) allows several organizations to learn a shared model from user data that only one organization is allowed to see. However, guaranteeing DP requires adding noise to gradients that scales with update dimensionality. While low-rank adaptation (LoRA) is a popular method for parameter-efficient fine-tuning, it is inherently sensitive to DP noise due to the multiplication operator. This noise amplification leads to significant utility degradation, hindering practical deployment in enterprise settings. To overcome this, we propose FedAnchor. Every round, FedAnchor captures compressed information using one of the LoRA matrices A and B, and then decompresses this to the complete adapted weight matrix via singular value decomposition (SVD) at the server. This relieves the model from DP noise exacerbation while maintaining full learning capacity. Experiments across multiple benchmarks demonstrate that FedAnchor matches the strongest baseline in non-private settings and outperforms it by 4.38% under stringent DP constraints (𝜖 = 3). At scale, FedAnchor maintains communication efficiency, computation robustness, and utility under strict privacy budgets, meeting the demands for enterprise-level cross-silo deployments.

Industrial post-training is a brownfield regime. Teams inherit a deployed checkpoint and must land targeted improvements under fixed compute and mixture budgets without regressing the rest. The maintained artifact is increasingly **dataware**: behavior governed by a curated post-training mixture, updated via bounded mixture patches rather than clean-slate retraining. From an industrial code-generation improvement effort, we offer a maintainer’s perspective on why this work is hard in practice, distilling three recurring challenges, **zero-sum mixture design**, **yield as the binding metric**, and **end-to-end integration under uncertainty**, and arguing that progress depends less on one-off recipes than on an engineering discipline for programming dataware. In our case study, interventions that raised the conversion of teacher distillation into usable training data increased accepted supervision by **2.84×** while using the same solution teacher and four solution attempts per candidate problem. In our primary evaluation, the yield-engineered patch improved CodeForces pass@1 by **+2.59** points (+3.11 pass@3) and held-out LiveCodeBench v6 pass@1 by **+6.11** (+8.05 pass@3), all statistically significant across 16 stochastic evaluations of each benchmark from one fixed checkpoint per condition, with internal AIME and MATH regression suites within tolerance

Knowledge graphs (KGs) are increasingly used as structured context for Large Language Models (LLMs), but industrial KG-RAG systems often need to integrate public and domain-specific KGs constructed from heterogeneous databases. This integration relies on Entity Alignment (EA), where lexical matching alone is insufficient under predicate-name variation and incomplete local neighborhoods. We address EA for KG integration by constructing a pairwise EA dataset and proposing two complementary modules: Predicate Importance Estimation (PIE) and Decoupled Rationale-Score Distillation (DRSD). PIE is a compact embedding-based approach that removes the subject information from each 1-hop triple, encodes the resulting subjectless triples, and aggregates them with learnable predicate-importance weights to build predicate-aware entity embeddings. DRSD trains a distilled small language model (SLM) with pseudo-answers produced by a teacher LLM through distinct prompts. By converting binary EA labels into text-based supervision and decoupling confidence-score estimation from label-consistent rationales, DRSD enables the SLM to learn task-specific reasoning while retaining a less label-biased confidence signal. Experiments show that PIE and DRSD improve EA classification. Moreover, because DRSD decouples confidence-score estimation from the decision, a division between the two flags an uncertain prediction for human review, thereby enabling a practical discrepancy between automatic acceptance and human-in-the-loop verification.

In specialist industrial imagery the dominant cost is maintenance under domain shift: labeling and fine-tuning must be redone on every new field, and proprietary imagery often cannot leave the operator’s environment. For geological drill-core fracture segmentation we release CoreFrac, to our knowledge the first open pixel-level benchmark on drill-core photographs: 200 expert-annotated full-resolution images with an expert-agreement ceiling (soft-F1 0.86). We retarget a frozen concept segmentor (SAM 3) by evolving only its natural-language prompt — no gradient updates, just a small labeled probe. Under a source-grouped protocol the evolved prompt reaches 0.545 positives-only soft-F1, beating rank-8 LoRA at the same 96-patch cap (0.487; paired +0.058, 95% CI [+0.020, +0.094]) and the best of eight expert prompts by 2.7×. Under a controlled domain shift, re-evolving on a small probe lifts a 0.622 → 0.409 drop back to 0.534. Its masks distill into a small on-premises U-Net (7.8M parameters, 3.8 ms/patch) that retains ≈92% of the teacher’s positives-only soft-F1 (0.501) at ≈1700× lower latency than *unbatched* SAM 3 — a cheaper, leak-free deployment. Code and data are released.

GraphQL has emerged as a widely adopted interface for accessing heterogeneous enterprise data. However, developing Natural Language to GraphQL (NL2GraphQL) systems remains challenging due to the need to rapidly adapt them to proprietary GraphQL schemas.

We present AutoGraphQL, a Bring-Your-Own-Schema framework for developing and evaluating NL2GraphQL systems. Given a GraphQL schema, AutoGraphQL synthesizes executable GraphQL queries, generates aligned natural language utterances, and validates them through schema and execution checks to automatically construct high-quality NL2GraphQL datasets. The framework additionally supports human-in-the-loop curation and an end-to-end workflow where the generated datasets are used to fine-tune and benchmark models.

Using AutoGraphQL, we construct a corpus of 17,621 NL2GraphQL instances spanning 173 GraphQL schemas and seven major query categories. Empirically, we show that fine-tuning open-weight models on AutoGraphQL-generated datasets substantially improves NL2GraphQL performance. Notably, the best fine-tuned 8B-parameter model achieves 42.57% benchmark accuracy, approaching Claude Sonnet 4.5 at 44.23%, while outperforming it on several GraphQL query categories. These results indicate that AutoGraphQL enables compact open-weight models to approach frontier-level performance on schema-grounded GraphQL generation. To support further research and benchmarking, we are releasing the training and test datasets under the MIT License. The dataset is available at https://nl2graphql-bench.github.io/.

AI systems increasingly answer scientific questions about health, safety, and the environment. But most retrieval-augmented generation systems are tuned to provide factually correct, on-topic answers rather than to help non-experts understand what those answers mean for their lives and decisions. We focus on consequential scientific questions whose results directly shape people’s lives and study them through a public water-quality communication system, where residents and community leaders interpret the findings and choose actions. Their experiences show that on-topic answers can still be insufficient without explanation and context and that the emotional weight of risk information cannot be ignored. Our system first classifies each question by reasoning type (for example, causal versus policy-based), then generates follow-up questions to identify missing evidence and retrieve it. One component clearly distinguishes between what is known and what is uncertain, while another rewrites scientific details into accessible language, using persona-based styles, such as a caring neighbor or an administrative official, to adapt tone and readability. Ablations on over 160 questions show that the system uses an order of magnitude less context and, in several configurations, improves human-rated completeness. A completeness metric co-designed with community members and a fine-tuned learned judge reveal that standard relevance scores explain about 1% of variation in human completeness ratings, and even the tuned judge only moderately aligns with humans, indicating that completeness is a distinct human-centered objective that current metrics do not reliably capture.

Multimodal large language models have demonstrated strong defect recognition capability in industrial anomaly detection. However, in lithography review, merely determining whether an image contains a defect is insufficient for engineering inspection; models must also understand defect morphology, spatial location, and the potential causes supported by visible evidence. To this end, this paper proposes LDU-Bench, a multi-task multimodal benchmark for lithography defect understanding. Constructed from real lithography and integrated-circuit review images, LDU-Bench decomposes the review workflow into four independent tasks: defect triage, morphology recognition, coarse localization, and image-conditioned cause analysis. It systematically evaluates models using task-level metrics, diagnostic readouts, and the Lithography Closure Score (LCS). Experimental results show that although existing MLLMs can perform defect triage relatively reliably, this ability does not stably transfer to downstream review stages. Morphology alignment, effective localization, and evidence-to-cause mapping remain the major bottlenecks. Further diagnostics indicate that this capability break is not a fluctuation of a single metric, but reflects insufficient structured understanding across semantic levels. Overall, LDU-Bench provides a quantifiable and diagnostic unified platform for evaluating the usability, failure points, and capability boundaries of industrial MLLMs in lithography review chains.

Matching users to interest categories at scale is central to personalized shopping, but the task is challenging in large e-commerce platforms, where label spaces continually evolve and user-interest signals are sparse and long-tailed. Autoregressive language models are appealing because their world knowledge and semantic priors over descriptors generalize across extreme label spaces and accommodate multiple valid label assignments. Yet under teacher-forced fine-tuning, inference-time predictions become part of the conditioning context: early errors steer later outputs toward co-occurring labels, over-generating near-correlates and missing unrelated true interests. We present MERIT, a framework for user-interest propensity modeling that mitigates this exposure bias through a self-correction objective. A permutation-invariant multi-target loss over shuffled mixtures of gold and mined hard-negative labels exposes the generator to erroneous prefixes while preserving the efficiency of teacher-forced training. This training objective concentrates supervision at classification positions, yielding propensity-aligned hidden states powering a lightweight scorer for bidirectional retrieval (interests for users and users for interests). On a proprietary e-commerce dataset with 250k+ interest categories, MERIT improves global recall by at least 11.9% and average Hit@k by 6.1%. In production A/B tests, it achieves +0.26% gain in user conversion.

Enterprise agents increasingly operate in workspaces, read heterogeneous files, use tools, and deliver business artifacts. We introduce EnterpriseClawBench, an internal enterprise agent benchmark constructed from proprietary agent sessions. Starting from a large archive of real workplace sessions, the EnterpriseClawBench pipeline produces 852 reproducible tasks with recovered fixtures, rewritten prompts, role classes, skill subclasses, hard rules, and semantic rubrics. Because the data contain internal enterprise content, the benchmark data are not released; the reusable contribution is the construction and evaluation protocol. On a manually audited 120-task Lite subset, the best score is only 0.663 for Codex with GPT-5.5. The results show that enterprise agent evaluation must report harness–model combinations, artifact delivery, visual quality, cost, runtime, and skill-transfer behavior rather than a single score.

Large language model (LLM)-based judges are increasingly used to scale the evaluation and optimization of open-ended medical dialogue systems, making their reliability critical in this high-stakes setting. However, existing medical benchmarks mainly assess their medical knowledge, while general judge benchmarks lack clinically grounded consultation criteria. To address this gap, we introduce Med-GRADE, a benchmark for evaluating LLMs as medical dialogue judges. It contains 2,076 real-world patient–LLM dialogues annotated by 56 physicians with a 23-item rubric adapted from Mini-CEX, covering Medical Interviewing (MI), Humanistic Care (HC), and Diagnosis and Treatment Management (DTM). Experiments on 17 LLMs show that current LLM judges achieve only moderate agreement, with the best model reaching 64.89% Macro-F1 and 68.28% Hamming Accuracy. Besides, more complex prompts may even harm performance: CoT disrupts parsing, while few-shot and self-refine introduce item-level instability. Data and code are available at https://github.com/jyd777/Med-GRADE for reproducibility.

Context engineering has emerged as a primary lever for improving AI systems without parameter updates. Recent work showing that textual gradients do not function as real gradients motivates treating automatic prompt optimization (APO) as black-box search. We introduce SPO (Stochastic Prompt Optimization), a framework for stochastic search over prompt space, and compare three strategies of increasing sophistication: error-informed random search, a genetic algorithm with evolutionary operators, and SAGE (SPO via Agent-Guided Exploration), a multi-agent pipeline with diagnostic code execution. Across three benchmarks, no single strategy dominates; effectiveness depends on the interaction of landscape structure with error type. We further deploy SAGE on a mental-health chatbot under a continuous optimization paradigm, where it compounds eight cycles of individually-noisy A/B tests into a statistically robust gain in next-day retention. We argue that coupling qualitative diagnosis with quantitative validation is what makes agentic optimization effective for open-ended task-oriented dialogue.

Agentic language models solve tasks through reasoning, tool calls, observations, and terminal answers, exposing rich process information beyond the final answer. For practitioners, knowing which process patterns lead to success or failure is essential to diagnose agent errors and build better process reward models without relying on final-answer supervision. However, existing analyses often evaluate only final outcomes or, when examining traces, flatten them into homogeneous step sequences, fragmenting each reasoning round into disconnected steps. Our key insight is that an agent’s reasoning, action, and observation within a round are better treated as typed segments of a single underlying process state, rather than independent steps. We propose MULTI-VIEW-RA, a reasoning abstraction that represents each round through two complementary views: accumulated state and within-round change. Using unlabeled clustering and binary outcome labels, we apply MULTI-VIEW-RA to two practical use cases: diagnosing failures by surfacing the process states that separate correct from incorrect trajectories, and providing a transition-level correctness signal that is a candidate source of process supervision. Across Math, HotpotQA, and FEVER with multiple Qwen and Llama models, MULTI-VIEW-RA outperforms step-level explainability baselines in cases with sufficient success rates, with particularly strong gains on longer, tool-rich traces.

As Large Language Models (LLMs) continue to scale both in size and capabilities, their proficiency in the Arabic Language has seen significant advancement. However, a critical gap remains: the extent of their factual knowledge and cultural sensitivity to the diverse Arabic-speaking world remains largely underexplored. Current evaluation metrics often focus on translation or generic reasoning, failing to capture the rich historical, social, and regional nuances inherent to Arabic culture. In addition, most benchmarks rely on heavy work, with human intervention in some steps, making the evaluation of knowledge coverage expensive and slow. To address this deficiency, we introduce AraDynFact, a novel dynamic evaluation framework designed to rigorously assess the factual Arabic knowledge embedded in LLMs. Unlike static benchmarks, AraDynFact employs a dynamic approach to extract factual information and generate rich and answerable questions in a fast and automatic way. We apply AraDynFact to Arabic Wikipedia and audit the performance of several state-of-the-art models, ranging from Arabic-centric specialized LLMs to high-resource general purpose LLMs. In addition we found high degree of correlation with existing, hand-crafted Arabic-centric benchmarks, confirming the potential of our dynamic approach.

Low-Rank Adaptation (LoRA) has become a popular method to adapt pre-trained models to specific domains. Applied to encoders in a multi-tenant service, it lets each tenant specialize a shared model for classification, retrieval, or ranking. Existing multi-tenant LoRA serving systems target autoregressive decoders, and for encoders the available tooling supports only per-request adapter swapping, whose cost grows with the number of registered adapters. We observe that, unlike decoder generation, the encoder forward pass is compute-bound and dominated by the shared base projections, so per-tenant adaptation can be expressed as plain batched matrix multiplication with no custom kernels. We present LateFuse, a pure-PyTorch system that serves many different LoRA-adapted encoders in a single shared forward pass on one GPU. LateFuse fuses adapters late, in activation space: the base projections run once over the mixed-tenant batch, while per-sample low-rank deltas and per-tenant classification heads are applied as batched matrix multiplications. Tenant weights are never merged into the shared base, ruling out cross-tenant contamination by construction. Our evaluation shows that LateFuse serving latency stays flat as the adapter pool grows to tens of thousands on a single GPU, and that it achieves 5.6–21.2x higher throughput than HuggingFace PEFT’s native mixed-batch API and up to 521× over per-adapter swapping, while registering new adapters over 299x faster.

Large language models (LLMs) are increasingly being adopted in clinical practice, e.g., in medical diagnostic systems. However, their reliability is still a concern, as LLMs are prone to hallucinations and could lead to misdiagnosis, inappropriate treatment decisions, and potential harm to patients. To mitigate these risks, LLM-based medical diagnostic systems should incorporate safety guardrails capable of detecting suspicious predictions and flagging them for human review or further clinical assessment. In this work, we employ uncertainty quantification (UQ) as a safety mechanism within an enterprise diagnostic system. We evaluate 16 UQ methods and propose two task-specific modifications tailored to the medical diagnostic workflow. First, we show that methods based on semantic similarity between generated samples substantially benefit from replacing a general-purpose NLI model with a domain-specific model. Second, we adapt a supervised UQ method that relies on LLM internal states for the detection of erroneous generation from a black-box enterprise LLM accessible only through a text-generation API. Our approach leverages the internal states of an open-source, lightweight proxy white-box model. We demonstrate that this approach outperforms all other UQ methods in the considered setting.

LLM-based voice agents are routinely cleared for deployment using synthetic audio regression suites, with engineering teams treating augmented commercial TTS as a safe proxy for real human callers. This paper presents the first systematic approach to quantify the profound behavioral drift that occurs when agents optimized on simulated audio are deployed in the real world. We demonstrate that the assumption of acoustic equivalence fundamentally fails: real production audio triggers 2-13 times more VAD misfires than synthetic data across Silero, WebRTC, and TEN-VAD, while simulated environments cause text-grounded turn detectors to artificially flip decisions 33% of the time, compared to just 3% in reality. A paired four-vendor TTS calibration sweep proves this misranking is structural: on every front-end component, optimizing thresholds for synthetic audio actively degrades real-world performance. To prevent teams from flying blind, we introduce a novel set of signal-layer metrics and an open-source framework to audit any simulator prior to adoption. Rather than testing the agent, our method tests the simulator against real target-domain interactions. By identifying where a simulator’s acoustics deviate from reality, teams can definitively know which front-end tuning decisions are safe to trust.

Complaint investigation in retail banking requires handlers to synthesise evidence from heterogeneous enterprise systems into policy-aligned summaries. We present a production-deployed tool-augmented LLM agent at a major UK retail bank that assists this process under human review. Given a complaint record, the agent queries 16 typed, schema-validated tools and returns an investigation summary with a source-cited chronological evidence timeline. We evaluate the system using SME assessment, LLM-as-a-Judge monitoring, rework measurement, and downstream impact testing. On 187 paired complaints, SMEs judged the agent to match or exceed historical human-written production investigations in 90% of cases across faithfulness, relevance, completeness, and coherence. Median investigation time falls from 30 minutes to 23 seconds. Operational controls include daily automated monitoring, pre-defined thresholds, hallucination flagging, and periodic human audit. These results demonstrate that constrained single-agent LLMs with typed tools are viable for scalable, auditable complaint investigation in regulated environments.

Cultural reasoning lies at the core of many emerging AI applications, yet existing benchmarks provide limited insight into how well models understand and contextualize cultural artifacts. We introduce ExhibitionBench, the first benchmark that frames exhibition curation as a high-level evaluation testbed, comprising 300 institution-derived exhibition and thematic collection records, 23,658 artifacts, and three interconnected tasks that evaluate complementary aspects of cultural understanding and reasoning, translating them into user-perceivable utilities. Extensive experiments across 29 systems reveal a significant gap between frontier LLMs and expert-level competence, particularly on tasks that require collection-level coherence and thematic understanding. Performance gains from scaling, prompting, and multimodal inputs are limited and highly model-dependent, whereas tailored model optimizations can equip open-weight models with better capabilities that support on-premise deployment. These findings highlight the difficulty of reasoning over cultural collections and provide a practical benchmark for developing more culturally grounded and reliable AI systems. The full project is released at https://github.com/mengze-hong/ExhibitionBench.

Large language models (LLMs) often follow conflicting reasoning paths on difficult tasks, creating uncertainty that is especially dangerous in clinical settings where errors can inform incorrect diagnoses or treatments. Standard self-consistency (SC) improves accuracy by voting over multiple reasoning chains, but it does not resolve the contradictions that produced the uncertainty. We propose Contrastive Adjudication via Selective Triggering (CAST), a lightweight inference-time method that detects disagreement across chain-of-thought samples and triggers contrastive adjudication only when model confidence falls below a threshold. Instead of re-asking the same question, CAST extracts the strongest argument for each competing reasoning path, presents them side by side, and asks the model to resolve the conflict definitively. Across multiple model families and three medical benchmarks, including HealthBench, a physician-rubric clinical task benchmark, CAST outperforms standard SC, compute-matched SC, SC-Max (with the maximum possible number of calls, usually 13), and the pooled CASC variant (which votes across all pre- and post- adjudication votes) across all datasets. Gains are largest on uncertain questions, reaching up to 29% over SC. These results, coupled with selective use of model calls, make CAST practical for clinical deployment where models must be accurate, efficient, and uncertainty-aware.

Large language models are increasingly used in oncology applications, but their predictions are often weakly grounded in explicit medical structure. We present TRACE, a deployable tree-relational enhancement framework for oncology LLMs. TRACE separates expensive offline structure learning from lightweight online inference: oncology concepts and relations are organized into an updatable tree-relational structure, refined using LM-loss-derived evidence, and retrieved at inference time as compact prompt evidence. This design supports task-adaptive evidence selection without requiring supervised labels in the zero-shot setting. Across ten oncology classification tasks and one MedQuAD CancerGov QA benchmark, TRACE improves both label-free evaluation and supervised fine-tuning. Additional analyses show that TRACE improves over vanilla RAG and generic GraphRAG, remains useful under leakage-controlled METABRIC inputs, and produces interpretable evidence paths aligned with clinical reasoning. These results suggest that explicit, updatable medical structure is a practical path toward more accurate and auditable oncology LLM deployment.

Economic sentiment plays a central role in shaping perceptions of economic conditions, influencing economic behavior, and policy decisions. Recent advances in large language models have made large-scale sentiment extraction from news increasingly feasible, driving its growing use in economic analysis and forecasting systems across national public-sector institutions. Existing approaches scale mainly through automated or weakly supervised labeling, sacrificing expert alignment and domain nuance, and treat sentiment in isolation, overlooking complementary annotations, such as topic, needed to capture structured economic signals for downstream forecasting. To address these limitations, we introduce a human-annotated dataset of Italian economic news labeled by domain experts for sentiment and topic, and develop a statistically validated pipeline that uses few-shot prompting of commercial large language models to expand coverage while preserving expert alignment at scale. We fine-tune multiple open-weight models on the resulting corpus, yielding language specific annotation models that maintain high expert agreement across architectures and satisfy the reproducibility and governance requirements of national public-sector institutions. Starting from 800 expert-labeled articles, the pipeline produces approximately 9,200 additional expert-aligned labels and trains local models that pass the Alternative Annotator Test for both sentiment and topic classification. A subset of the annotated dataset is publicly released to support further research.

Financial decision-makers face more information than they can directly inspect, making context compression necessary. Yet when large language models (LLMs) compress financial source material, they can alter the investment judgment supported by the original source. We frame this problem as information fidelity: compression loses fidelity when it changes the decision induced by the source. In agentic systems, such losses may recur across intermediate steps and amplify throughout the decision process. Across financial filings and earnings-call transcripts, we find that LLM-based compression can produce fluent and factually plausible compressed contexts that nevertheless alter downstream decisions. We analyze two diagnostic patterns associated with fidelity loss: decontextualization, where salient evidence is retained but separated from the caveats and contextual qualifiers needed for correct interpretation, and model dependency, where different compressors expose different views of the same source. We then propose Agentic Context Compression, which generates multiple candidate compressions and audits their disagreements against the original source. Our results suggest that financial compression should be evaluated not only by efficiency or factuality, but also by its ability to preserve decision-relevant context.

Wearable assistants require episodic memory over egocentric video, yet current vision-language models face bounded frame budgets, growing visual-token costs, and long-context retrieval failures. Under these practical constraints, we study whether textual captions can serve as reusable episodic memory. We define the Episodic Memory Video Caption QA task and introduce CapMem, a human-annotated benchmark with 75 videos totaling 33.7 hours, and 1,000 multiple-choice questions across 16 scenarios. On long videos (>20 min), full-coverage CaptionQA with 30s and 60s caption windows outperforms direct VideoQA for 10/12 and 8/12 models, respectively. On the same video subset, a matched-frame control across six Qwen models retains mean accuracy gains of 3.22 and 2.55 points, respectively. Our caption-guided retrieve-and-verify harness further improves accuracy by up to 5.3 points. These results support the effectiveness of caption memory for episodic reasoning over long egocentric video.

Fraudulent messages sent via Short Message Service (SMS) are increasingly obfuscated to evade cost-conscious classifiers in production systems. In Chinese SMS, attackers can exploit a wide range of carefully crafted obfuscation strategies to hide risk-bearing phrases while preserving human readability, making direct classification brittle under real-world latency and throughput constraints. We propose ReCAST, a Restoration-aware Cascaded Stage-wise Training framework for robust obfuscated Chinese SMS classification. ReCAST distills a large teacher model’s de-obfuscation ability into a smaller deployable student model by supervising obfuscated span detection, obfuscation type prediction, and text restoration, and then uses the restoration-aware student for downstream risk classification. Experiments on an internally constructed real-world Chinese SMS benchmark show that ReCAST substantially improves classification performance over directly trained baselines under obfuscation. The results suggest that restoration-aware distillation offers a practical path toward robust SMS risk classification with smaller deployable models under production-oriented constraints.

Generating Business Process Model and Notation (BPMN) process models with LLMs is costly at the serialization layer. Standard XML and JSON formats consume hundreds to thousands of output tokens per model, creating severe latency and cost bottlenecks for interactive modeling assistants. We present BPCF (Business Process Compact Format), a compact, indentation-based notation designed specifically for LLM emission. BPCF-Full is lossless with respect to the structural constructs found in the PMo benchmark. At 221 tokens per model on average, BPCF-Full is roughly half the size of a Mermaid encoding, 8× smaller than verbose JSON representations (PME JSON) and about 29× smaller than BPMN XML. We evaluated nine BPMN representations across five LLMs and found that BPCF is the most token-efficient format by a wide margin, delivering the lowest cost and latency. Although exact accuracy rankings vary across models, BPCF offers a highly viable accuracy-efficiency trade-off for production systems, successfully modeling complex BPMN constructs where other compact formats fail. In the full-expressivity arm, BPCF-Full+Strict scores 0.469/0.472 on GPT-4o/Gemini at 148–155 visible output tokens. We publicly release our grammar, a dependency-free parser, and the full evaluation pipeline.

News articles often contain omitted temporal expressions, such as day-only or month-only mentions, which must be interpreted with reference to the publication date. When such articles are indexed or processed as standalone text in search and retrieval-augmented generation (RAG) systems, these omissions can cause temporal mismatches and unstable interpretation by large language models. We focus on reproducing omitted temporal expressions as concrete dates or intervals using the publication date as external context before the articles are indexed for search and RAG applications. Specifically, building on established temporal-expression extraction and normalization techniques and informed by a manual analysis of Japanese news articles, we propose jaROTE, a rule-based pipeline for Japanese news. Experiments on two news corpora demonstrate that jaROTE achieves high performance, and remains competitive with LLMs while providing a fast, low-cost pipeline. We further show that temporal reproduction improves time-constrained lexical retrieval, demonstrating the practical value of publication-date-grounded normalization for Japanese news retrieval.

Data-sovereignty regulations increasingly require public institutions to deploy open-source, on-premise LLM agents that chain multiple tool-calls across live government APIs. However, open-source models consistently underperform in this multi-step setting, and no existing benchmark measures the gap. We introduce the Korean Open Public API Benchmark KOPA-Bench, comprising 145 real-world tasks on which open-source models underperform. To close this gap, we present EDGE, an Execution-grounded Dynamic Graph for tool-calling data synthEsis driven by live execution. EDGE builds a graph of how each tool’s output can feed another’s input, keeps only the links that succeed when actually called against the live APIs, and traverses these verified links to synthesize executable multi-step trajectories. Fine-tuned via GRPO on the resulting dataset, our 9B model nearly matches the untuned 27B model from the same family, improving substantially not only on KOPA-Bench but also on the BFCL benchmark.

Technical-service LLM agents are entering production workflows, where value depends on whether engineers adopt generated replies. Service tickets hide decision logic, contain noisy single-reference responses, and make reward evaluation costly, making standard post-training brittle. Existing post-training and LLM-as-a-Judge approaches improve grounding or feedback, but do not jointly model latent decision logic, response diversity, and reward cost. We address this gap by coupling latent logic augmentation, robust noise reduction, and hybrid reward modeling. The framework augments supervised fine-tuning data with Planning-Aware Trajectory Modeling and Reasoning Augmentation, builds dual-filtered Multiple Ground Truths, and trains the policy with a hybrid reward that combines a Reranker with an LLM-as-a-Judge. On real Cloud technical-service tasks, the adapted Qwen3-4B achieves the highest Multi-ECS (0.441), lower reward cost, and the highest production adoption rate (46.63%).

General-purpose large language models (LLMs) that rely on in-context learning do not reliably deliver the scientific understanding and performance required for drug discovery tasks. Simply increasing model size or introducing reasoning tokens does not yield significant performance gains. To address this gap, we introduce the MMAI Gym for Science, a one-stop shop molecular data formats and modalities as well as task-specific reasoning, training, and benchmarking recipes designed to teach foundation models the ”language of molecules” in order to solve practical drug discovery problems. We use MMAI Gym to train an efficient Liquid Foundation Model (LFM) for these applications, demonstrating that smaller, purpose-trained foundation models can outperform substantially larger general-purpose or specialist models on molecular benchmarks. Across essential drug discovery tasks - including molecular optimization, ADMET property prediction, retrosynthesis, drug–target activity prediction, and functional group reasoning - the resulting model achieves near specialist-level performance and, in the majority of settings, surpasses larger models, while remaining more efficient and broadly applicable in the domain.

Synthetic tabular data lets organisations share, rebalance, and stress-test sensitive in-house tables-but only if the result is actually fit for the downstream task. Yet a table that matches the real distribution on aggregate statistics can still be unfit for its downstream task: it loses the rare cases that matter, is not optimised for the end task, and silently breaks the deterministic relations that domain data must obey. We present TabAutoSyn, an end-to-end, generator-agnostic pipeline that turns any base generator into deployment-ready synthetic data through four automated stages: generate, verify, augment, and curate. A multi-agent LLM module discovers functional dependencies and domain constraints from the real table and repairs or removes violating rows; a divergence-minimising stage rebuilds under-represented tails; and a curation stage provably never lowers a chosen downstream score. Across 22 public and proprietary datasets and eleven generators spanning five model families (GAN, VAE, diffusion, language-model, and graph), TabAutoSyn delivers the best train-on-synthetic utility and the highest validity while preserving privacy. Because it is generator-agnostic, it attains these results on a commodity GPU at a fraction of the cost of the strongest LLM-based baselines, and it needs no per-dataset tuning-making it practical to run on sensitive, in-house data.

Masked diffusion language models reuse one denoiser across many steps, so post-training quantization (PTQ) must balance predictive quality, serialized storage, and runtime memory. We evaluate LLaDA-8B-Instruct and run transfer checks on Dream-7B, MDLM-OWT, encoder-only masked LMs, LLaDA-MoE, and LLaDA2.0-mini. The 24-text calibration set is distinct from the main n=500-example-per-task evaluation. Three findings emerge. First, AWQ-4 is the closest tested 4-bit quality point at the evaluated 1.1–2.3 percentage-point resolution, although ARC-Challenge shows a detectable loss. TurboLDM-(3+1) instead provides the smallest serialized checkpoint (4.82 GiB, 3.55 effective bits) while losing quality, making it a storage-compression and diagnostic result, not a live-memory result. NF4 alone reduces peak VRAM from roughly 15 GiB to 6 GiB in our tested PyTorch/bitsandbytes runtime; this is a runtime-specific outcome, not an inherent format advantage. Second, individual attention projections are inexpensive to quantize, whereas jointly quantizing the attention block is damaging. Third, more denoising steps partly recover GSM8K loss but not multiple-choice accuracy.

The widespread use of large language models (LLMs) has increased the risk of leaking personally identifiable information (PII), making PII detection increasingly important. PII detection must process large text volumes efficiently, but labeled PII text is difficult to collect, share, and redistribute for privacy reasons. We aim to develop rubric-guided contrastive data synthesis for training lightweight PII text detectors from a few labeled seeds and to determine, through comparative experiments on PII benchmark datasets, whether the proposed method improves PII text detector classification performance. From about ten labeled examples, our method induces explicit criteria for distinguishing the two labels (rubrics), generates contrastive Sensitive/Non-sensitive sentence pairs, verifies and deduplicates them, and trains RoBERTa. Experiments on SPeDaC1 and SPY with Qwen3.5-9B and Gemma-4-26B-A4B showed that our method outperformed four baselines in all four settings and significantly improved Macro F1 over the Self-Instruct-style baseline in three of four settings. Overall, we developed rubric-guided contrastive data synthesis for training lightweight PII text detectors from a few labeled seeds and confirmed through comparative experiments on PII benchmark datasets that the proposed method improves classification performance. Our code is available at https://github.com/ekunish/emnlp2026-code.

Multi-agent debate systems incorporate retrieval-based verification to ground agent reasoning with evidence. However, this approach assumes the availability of data and the informative intent of the source text. These conditions are frequently unmet in venture capital, particularly in early-stage startup evaluation. In this industry, evidence is predominantly private or unavailable, and pitch decks use persuasive rhetoric that prior research shows can shift agent conclusions. To measure how agents navigate these conditions, we evaluate how claim verification alters multi-agent debate using a stratified sample of deals from a real venture fund. We find that verification changes the agent’s final investment decision but not its reasoning. Furthermore, agents are insensitive to which claim a verification label is attached to. Reassigning labels to the wrong claims lowers the decision to the same extent. This indicates that the primary bottleneck in automated screening is fundamental claim verifiability rather than retrieval quality.

Evaluating LLM-powered AutoML agents for enterprise data science requires more than final leaderboard scores: agents must deliver reproducible, leakage-free, protocol-valid workflows on organization-specific data before deployment. We introduce GRACE-DS, a Guarded Reward-guided Agent Correction Environment in Data Science for evaluating tabular machine learning agents in real-world-like conditions. GRACE-DS stages the full workflow—planning, data inspection, feature engineering, modeling, validation, code repair, and final submission—inside an isolated sandbox where evaluator-owned labels and hidden tests remain private. Hidden executable validators track both predictive progress and process reliability, including leakage avoidance, reproducibility, validation discipline, correction behavior, reward alignment, and terminal submission validity. Based on 7,000 episodes spanning 8 LLMs, 10 tasks, 15 regimes, ablations, reward-optimization probes, and red-team settings, the structured flexible iterative regime (our approach) achieves the strongest end-to-end normalized hidden-test quality, outperforming single shot, unstructured agent, and restart baselines while improving protocol-valid completion. These results establish GRACE-DS as a controlled pre-deployment harness for diagnosing whether LLM-powered AutoML agents satisfy core enterprise execution constraints, on a deliberately search-free tabular sub-problem that isolates planning, correction, and validation discipline from expensive model search. An earlier version of this paper is available at https://arxiv.org/abs/2606.16000v2; code and artifacts are available at https://github.com/Alexx221x/GRACE-DS/tree/archive.

Attribute-value extraction is important for building high-quality product catalogs in e-commerce. Existing systems either rely on generative or tagging strategies for extracting the attribute names and their corresponding values. Tagging models directly label spans in the source, making them fast and grounded, but they cannot produce values that require reformatting or generate rare attribute names outside their label space. Generative models avoid these drawbacks by generating both attribute names and values token by token, but this auto-regressive nature makes them slow and limits attribute coverage. We introduce hybrid generative-tagging decoding, a novel model architecture that combines tagging source spans and open-ended generation. For frequent attributes whose values are copyable, a tagging head extracts values directly from the source. This lets the generative head focus on the remainder, rare attributes outside the tagger’s label space and non-copyable values, including implicit or normalized values. Integrated into vLLM inference, the hybrid decoder raises attribute coverage by 10% and extracts 63% more correct attribute-value pairs at no precision cost, while reducing latency by 44%.

Large Language Models (LLMs) often help users generate artifacts through iterative cycles of generation and revision in conversation. A challenge here is that, when users specify only a local change during revision, LLMs must instead identify the relevant dependencies and propagate the revision to all affected parts of the artifact. This paper studies this ability of LLMs on conversationally generated artifacts, where the artifact context and its dependencies may be embedded in the conversation history. Toward practical use, we also explore cost-effective test-time compute for this new setting. Specifically, we introduce a new benchmark for this setting, and evaluate nine revision methods, including sequential reflection and parallel sampling variants, using gpt-oss-20b/120b, gpt-5.4-mini, and qwen3.5-9b/27b/122b on the benchmark. The results show that baselines achieve accuracies of 68.3–93%, and the most cost-effective method is selecting from three parallel samples using either LLM-based or medoid selection, which improves accuracy by 2.2–9.7%. Our code and dataset are available at https://github.com/ntt-dkiku/llm-revision-propagation.

Enterprise agents face safety risks that cannot be determined from individual actions alone. Existing safety benchmarks largely evaluate action-local risks, overlooking failures whose safety depends on workflow state. In such state-dependent risks, the same action may be safe in one context but violate policy in another. Hence, we introduce LatentRiskBench for evaluating such state-dependent risks in enterprise settings. It pairs risky trajectories with benign counterparts that share the same terminal action but differ in latent policy-relevant states, requiring models to reconstruct workflow context rather than rely on action-level cues. We also introduce new metrics for state-dependent risk evaluation, including a suite of residual risk (RR) measures. Experiments on the benchmark show that action-level evaluation detects only 0.09 of state-dependent violations, whereas state-level evaluation reaches 0.96. Existing guardrail methods largely overlook such risks unless they are explicitly applied to workflow states. Case studies on a commercial deployed scenario further show that the safety metrics embedded in the benchmark provide a more realistic measure of safety for enterprise agents.

Detecting suggestions in reviews forces a trade- off: an LLM on every review is slow and costly; a cheap classifier is fast but its error rate is uncontrolled, measurable after the fact, not settable in advance. We present a certified triage layer: a calibration step computes ex- actly where a classifier may act alone so its error there stays below an operator-chosen tar- get α; the rest is escalated. On a 14,995-review gold set over 68 industry categories, natural- weighted to a ∼9.5% deployment positive rate, it auto-handles 74–93% of the stream at cer- tified error for α ∈ 2, 4, 5, 10%. Heuris- tic thresholding violates the α=5% target on 36–49% of draws and an in-expectation con- formal gate on 25–28%; only a betting certifi- cate bounds the violation rate by δ, making a strict 2% SLA rigorous at 74% order-free cover- age. Cost-sensitive training, not threshold tun- ing, lowers the certifiable error floor to 0.39%, and ∼200 target labels restore it under cross- industry shift. Of 13 open-weight LLMs (20B– 675B), 11 are worse detectors than our fine- tuned 435M encoder, supervision, not scale, is why (within-model control, §5.6), and the cer- tificate rides on either backbone. Escalation targets a human: on the deferred slice the best LLM oracle still errs at 21.5%.

Demand for personalized financial advice is growing, yet current LLM-based advisors often fail to provide consistent and specialized guidance. Simple persona prompts rarely specify how a financial advisor should reason and often drift toward generic recommendations. We propose Fund2Persona, a framework that builds financial-advisor personas from real-world fund disclosures and refines them through an actor–scorer–patcher loop. We test whether the resulting personas can predict held-out portfolio changes and produce commentary consistent with fund managers’ own explanations. They outperform generic baselines on both tasks. We further study two downstream diagnostics: market-scenario generation, where persona retrieval broadens plausible investment views, and multi-turn investor–advisor conversations, where matched personas give more specific and useful advice than a generic advisor. These results suggest that real fund data can bring manager-specific investment expertise to LLM advisors rather than merely changing an LLM’s surface style.

Recorded lecture videos, often enhanced with search and summarization features, are a standard study resource. However, students cannot easily ask course specific questions or verify answers against an instructor’s lecture. We report a semester-long deployment of VideoPoints platform with a retrieval-augmented chatbot that answers from course lecture materials and returns timestamped citations. The chatbot retrieves only from the active course, uses chapter summaries to guide transcript ranking, and returns clickable timestamped citations. Students used it for quick lookups and exam review. Across 833 messages, 70.5% included citations, none crossed a course boundary, and when no lecture evidence matched, the chatbot usually declined rather than answering. Among the users, citations were the most consistently useful feature, while practice-question generation was the strongest unmet request. We also evaluated the design on the real-world test split of EduVidQA, a public multimodal benchmark for lecture-video question answering. Our design improved correct-lecture retrieval by 6.3 percentage points over dense-only retrieval. Together, the results show that effective deployment depends on course isolation, supported citations, and alignment with students’ study practices.

Prompt-side playbooks can improve tool-using language agents without retraining, but their portability beyond the source setting is unclear. We study frozen playbook transfer under a shared distill–validate–transfer protocol. On ALFWorld, transfer is beneficial under controlled greedy decoding and, in one near-budget-matched comparison, distilled guidance outperforms five fixed demonstrations. On TAU2-Bench, a prespecified aggregate contrast supports a modest average matched-domain advantage, but global Holm correction retains only one of 135 route-level effects; the remaining grid provides descriptive evidence of compatibility-sensitive heterogeneity. On XBench-DeepSearch, one artifact–runtime pairing preserves useful first-try heuristics while producing repeated queries, delayed stopping, and substantial cost inflation after a context-runtime shift. Across benchmarks, transferred and target-derived playbooks both require target-side validation of success, termination, protocol compatibility, and cost. Frozen transfer is therefore a conditional cold-start option, not a reuse-by-default strategy or a universally preferable alternative to target-side redistillation.

Large Language Models (LLMs) have achieved strong performance across diverse NLP tasks, but deploying them in real-world scenarios raises concerns regarding safety, reliability, and computational cost. To mitigate the risk of harmful, unreliable, or unfaithful content, LLM pipelines increasingly rely on guardrails. Yet deploying multiple guardrails across different tasks and risk categories introduces significant computational and orchestration overhead. In this work, we propose a calibrated deferral-routing framework for deploying guardrails both accurately and cost-effectively. Our approach arranges guardrails as a cascade of increasing capability and cost: each input is first evaluated by a lightweight guardrail, while only uncertain instances are deferred to a larger, more capable, and more expensive model. We formalize this routing problem and study practical deferral-calibration strategies by applying post-hoc calibration, including isotonic regression, Platt scaling, and temperature scaling. We evaluate 18 guardrail models from four model families (i.e., Granite Guardian, Llama Guard, ShieldGemma, and GuardReasoner) across 10 harm-detection benchmarks. Our experiments show that: (i) all guardrail models are often poorly calibrated out of the box (ECE ranging from 6% to 28%); (ii) post-hoc calibration substantially improves reliability reducing ECE below 2.5% in most cases; (iii) cascade routing yields large cost savings with minimal accuracy loss, while monotone calibration primarily makes routing thresholds deployable rather than improving rank ordering.

Robust understanding of user input is a core requirement for voice assistants deployed in real-world environments. In practice, these systems encounter heterogeneous fallback situations caused by noisy audio input, transcription errors, ambiguous requests, incomplete utterances, or unintended activations. Existing systems typically respond with generic fallback messages, which do not resolve the underlying interaction failure and can degrade user experience. We study fallback handling in a deployed smartwatch-based voice assistant for general health support in everyday environments. Our analysis is based on six months of real-world usage data from more than 500 users, yielding a dataset of 3,030 anonymized, naturally occurring fallback-triggering utterances. We contribute (1) an operational taxonomy and the annotated VoxFallbacks dataset of these interactions, (2) a comparative evaluation of different models within a classification pipeline under practical deployment constraints, and (3) practical lessons for designing robust and cost-efficient fallback mechanisms. Results show that lightweight embedding-based classifiers outperform larger generative models on most classification tasks while requiring substantially fewer computational resources.

Advertising operations (AdOps) include compliance review processes to ensure that advertisements adhere to applicable laws, platform policies, and industry guidelines. In particular, ads must avoid issues such as overclaiming effectiveness or making definitive assertions about uncertain outcomes. However, these reviews are typically conducted by experts, making them an operational bottleneck. In this paper, we propose the automated ad text compliance verification task and present our classifier-based system. We introduce AdCompliance, a dataset consisting of real pre-submission ad texts annotated by experts. Our experimental results demonstrate practical classification performance, suggesting that non-compliant ads can be effectively filtered while maintaining a low false positive rate. These results support practical integration into AdOps workflows.

Large language models are increasingly capable in general, but their utility can remain modest in niche or understudied areas. One approach to address this limitation is to specialise existing models through additional training on target-domain corpora. In this work, we investigate such continued pre-training for adapting large language models to Swedish journalism, using a high-quality dataset that we curate from millions of news articles. To evaluate the adaptation efficacy, we also construct a novel domain-specific benchmark that covers six editorial tasks. Through full and parameter-efficient fine-tuning across two model sizes, we find that continued pre-training yields benefits in the target domain, but only when paired with experience replay to mitigate forgetting. We observe consistent enhancements in the models’ generation quality and factual knowledge, but not their proficiency in discriminative tasks. Exploring a training-free method to facilitate instruction following, we see further improvements, but exclusively for models trained with low-rank adaptation. Crucially, we demonstrate the importance of targeted evaluation in the adaptation process, as an existing Swedish benchmark largely fails to capture the models’ in-domain performance gains.

Re-identification methods serve as a concrete instantiation of the Motivated Intruder Test in privacy-preserving problems, and current state-of-the-art attack techniques are predominantly built upon Large Language Models (LLMs) which are also core components of robust pseudonymized methods. Therefore, it requires the rich training dataset. However, as research demands continue to grow while publicly available data for each domain remains limited and the cost of direct human supervision is high, a central question arises: how can a human-in-the-loop approach be integrated to generate high-quality data and improve the performance of LLMs in psedonymization. To address this limitation, we propose MOTIGAN1 - a gamified framework that transforms the motivated intruder test into a competitive three population tournament. In this game, human characters and AI characters compete to re-identify individuals from each essay. At every stage, players both commit their guesses and contribute to enriching the synthesized dataset. In our experiments, the synthesized dataset contributes to improving the performance of multiple models in several experiments (achieving up to 37.6% improvement on a well-established benchmark).

Intent classification deployments trade accuracy against retraining effort and inference cost. We study frozen generative reranking and budgeted large-model escalation, introducing Semantic-Candidate Adaptation (SCA): BGE top-5 retrieval followed by joint reranking with one representative training example per candidate. On CLINC150, aligned examples raise Qwen2.5-3B-Instruct from 68.18% label-only accuracy to 76.13% and Gemma-2-2B-it from 66.02% to 76.15%, without LLM parameter updates. CLINC150 wrong-example controls support the role of correct alignment, while BANKING77 and HWU64 provide transfer evidence. Broader comparisons clarify the deployment regimes: direct centroid retrieval reaches 80.71%, while a task-trained MPNet Energy baseline reaches 90.05%; always-on SCA is therefore not a universal replacement for direct or trained classifiers. A validation-selected low-margin SCA fallback provides only a marginal gain at an 8.62% realized test call rate. For real Qwen3.7-Max escalation on CLINC150, routing the lowest BGE top-1 retrieval scores reaches 82.11–82.89% accuracy at a 10% budget and 87.33–88.07% at 20%, with OOS F1 of 70.00–70.13% and 87.47–87.70%, respectively. Learned and token-confidence routing remain alternatives, but no policy dominates every evaluated point.

Logical anomalies in industrial inspection are rule violations: an object count is wrong, an item is misplaced, a component type does not match, or a required relation is broken. Existing anomaly understanding systems usually stop at diagnosis, answering whether an image is anomalous, where the anomaly is, or why it violates a rule. Reworkable inspection workflows require the next question: What should be fixed? We introduce an industry-oriented benchmark and training pipeline for rule-grounded corrective actions. The task asks a model to produce an operator-facing action that supports one of three downstream decisions: no action, rework, or reject/manual review. This setting exposes a Diagnosis-Action Gap: identifying an abnormal object does not imply knowing the valid, minimal action that restores normality. We construct from MVTec LOCO and from VID-AD, adding correction-oriented annotations that connect normal rules, visual evidence, repairability, corrective actions, and verification targets. The datasets support multiple-choice correction selection, structured JSON action prediction, and open-ended correction generation. We further train ReworkVLM, a Qwen3-VL-based model optimized with , which combines semantic correctness, trace-answer alignment, and minimal-edit rewards. Experiments on both benchmarks show that correction-oriented evaluation reveals failures hidden by diagnosis alone, including wrong action direction, unnecessary repair, and non-minimal edits, while ReworkVLM improves structured corrective success and open-ended correction quality over general and anomaly-specific VLM baselines.

As LLMs become everyday creative partners in software development, code is no longer a purely human artefact; it emerges from an increasingly hybrid human-AI process. Understanding who contributed which parts of a codebase has become a foundational question of creative attribution, touching on authorship, trust, and the integrity of co-created work. Yet existing detectors exceed 100M+ parameters, are too slow for practical deployment, and offer no insight into what distinguishes human from AI creative contributions. We present NanoDroid, a backbone-free detector with only 5.1M parameters that runs 20× faster than 125M-parameter UniXcoder (SOTA) and is fully interpretable: per-stream attribution scores explicitly expose which stylistic evidence (token statistics, transition patterns, surface style) distinguishes human from AI authorship in code. On the challenging 4-class DroidCollection benchmark (Human / Machine / Refined / Adversarial), NanoDroid achieves 90.77% weighted F1, within 2.6% of SOTA, on the CodeT-M4 benchmark, NanoDroid achieves SOTA performance, while providing the kind of transparent decision decomposition that black-box models cannot. Beyond a detection tool, we argue NanoDroid offers a lens for studying how human and AI creativity differ stylistically, and where they blur.

Sovereign AI deployments often require foundation models to operate under constraints overlooked by standard benchmarks: adherence to legislation, multilingual and multimodal grounding in local contexts, institution-specific safety behaviors, and on-premises deployment on power-constrained infrastructure. We present a production case study detailing the multi-stage engineering pipeline that spans trillion-token-scale continued pretraining, multi-billion-token alignment and post-training for a 123B-parameter natively multimodal foundation model, *Phoenix-VL 1.5 Medium*, optimized for Singapore public-sector use. On our novel localized evaluation suite and behavioral safety framework, the model substantially improves localized multimodal, statutory, and public-safety capability (a +0.118 average gain over the strongest baselines) while retaining broad general capability. We also share empirical analysis from a full production deployment currently serving over 30,000 users and processing ∼1 billion tokens over a two-month period. Ultimately, our findings demonstrate that sovereign AI is best approached as an end-to-end systems problem, where data curation, domain training, institutional alignment, and hardware layout interact and cannot be optimized in isolation.

Enterprises fine-tune language models on proprietary data that may later require removal due to privacy, contractual, or compliance obligations. Selective unlearning removes requested knowledge while preserving model utility, offering a practical alternative to full retraining, but existing methods treat the explicitly identified forget examples as the complete deletion scope. This is insufficient when target knowledge remains recoverable through paraphrases, aliases, or neighboring training examples. We propose GraphSU, a graph-guided controller that expands the deletion scope beyond forget seeds by constructing a weighted support-route graph, propagating deletion pressure through it, and applying graded forgetting strengths to high-risk neighbors. On TOFU and PISTOL with GPT-2 Medium and Llama-3.2-3B-Instruct, GraphSU achieves the lowest utility-feasible soft leakage across all deletion settings, reducing leakage by up to 49.5 percentage points over a matched seed-only baseline, demonstrating that effective enterprise unlearning requires controlling support routes, not just forget seeds.

Deep research systems iteratively plan, gather evidence, synthesize, and revise, making end-to-end latency a key bottleneck for interactive use. Almost all production systems are built on autoregressive (AR) language models and inherit left-to-right decoding in planning and long-form writing. We present DiffResearch, an open multi-agent deep research framework that places a diffusion language model (dLLM) at the core of every stage, and measure how a diffusion backbone changes latency and report quality. Holding the agentic scaffold, retrieval code, prompts, and evaluation fixed, we swap the backbone between an open diffusion model and a comparable AR model. Across 100 Deep Research Bench tasks, the diffusion backbone is roughly 6× faster end-to-end, driven by order-of-magnitude reductions in planning and writing. We further introduce dFast, a decomposition mode that reads intermediate denoising states and extracts subqueries before generation finishes, providing a 1.99× median speedup for decomposition in isolation with no loss of report quality. On the benchmark’s RACE rubric, dFast obtains the best overall score among the three configurations. Finally, in a public system-level leaderboard snapshot, an instance built on a stronger diffusion model ranks above several commercial systems. We release DiffResearch under an Apache-2.0 license.

Enterprise document security classification is a selective-decision problem. Rare high-risk labels require evidence and auditability, yet judging every document with an LLM is costly and adds model-specific variance to cases that precedents already settle. We introduce CARE (Cost-Aware Resource-adaptive Evidence), a bounded, fixed commit–verify–adjudicate controller that routes each case by precedent trust. It forms a zero-LLM evidence vote, commits when a learned reliability gate predicts low vote-error risk, verifies uncertain cases through verifier–evidence agreement, and spends heterogeneous panel judgment only on the residual conflicts where evidence and verification disagree. Across two contrasting evidence regimes that defeat different fixed baselines, the same controller selects route depth document by document. On WikiLeaks, where contrastive retrieval already supports strong performance without LLM calls, CARE preserves the zero-call evidence endpoint within bootstrap uncertainty while staying above every always-LLM decoder. It reaches 0.9824 Macro-F1 and 0.9624 SECRET F1, with only 44 of 2,863 documents reaching any LLM stage. This requires 55 invocations in total, 1.9% of a one-call-per-document budget. On DISC, where the rare TOP SECRET tail destabilizes sparse and LLM decoders, it gives the strongest row in our controlled comparison (0.8785 Macro-F1, 0.8444 TOP SECRET F1) using 496 physical model invocations against 789 for a same-evidence always-LLM policy, 63% of that budget, with only 81 documents reaching the panel. CARE thus turns security classification into an auditable policy for deciding when evidence is enough and when judgment is worth spending.

Large Language Models (LLMs) increasingly act as autonomous agents executing complex, long-running procedural skills. Existing agent runtimes maintain execution by continually appending observations, actions, and intermediate reasoning traces to an ever-growing conversation history, causing latency degradation and context-poisoning failures over long horizons. We present SKILL.state, a runtime architecture that replaces append-only conversational history with an explicit, mutable execution state. At each execution step, the model receives only the immutable skill specification, the current structured execution state, and the latest observation. Intermediate reasoning is discarded immediately after producing a validated state update, preventing prompt growth with execution history. Across diverse datasets, models, and execution environments, SKILL.state improves task accuracy while substantially reducing cumulative token consumption. Our results demonstrate that explicit execution state is an effective and architecture-agnostic abstraction for scalable long-horizon agent skills.

Multi-agent LLM results can conflate decomposition with roles, communication, parallelism, and extra compute. We study strictly ordered tool-use workflows of dependent state-changing steps, where segmentation may also support auditing and handoff constraints. We introduce Trace-GRAF, a granularity selector that aligns one single-agent trace to target workflow steps, converts order violations and unrelated drift into split pressure, and selects the smallest granularity on the minimum-pressure plateau. It executes balanced contiguous segments because boundary ablation finds no significant success difference from semantic segmentation; pressure-minimizing partitions characterize cuts. Across held-out cohorts spanning two benchmarks and three executors, fixed decomposition sometimes improves and sometimes reduces success, while greater granularity both rescues failures and regresses solved runs. When SA success is observable, the pressure route preserves successful runs by construction. Across validation resplits on TerminalWorld, the calibrated extension’s mean recovery is 2.3–3.0% for GPT-OSS, 0.0–0.1% for Gemini, and 2.3–4.9% for DeepSeek. Simple fixed fallbacks match much of the observed gain. Trace-GRAF therefore provides a risk-aware routing diagnostic with benchmark- and split-dependent gains.

Deploying large language models for legal question answering raises challenges that general-purpose leaderboards do not capture, particularly for low-resource languages and under hard operational constraints. We report on building and operating a retrieval-augmented (RAG) legal assistant for Uzbek that must run in two regimes: a managed cloud service that maximizes answer quality within a per-token cost ceiling, and an on-premises deployment for clients whose legal data may not leave their infrastructure, restricting us to open-weight models on limited local hardware under latency constraints. Because no evaluation existed for this setting, we build two domain benchmarks: a retrieval benchmark of 178 expert-annotated legal queries with gold provision spans, and an end-to-end benchmark of 504 expert-curated question–answer pairs scored by an LLM judge whose ratings we validate against human judgments and against an independent-family judge. Applying these benchmarks under each regime, we find the open-versus-proprietary gap is small and cheaply closed by fine-tuning. Therefore, we train UTE-1, which is a state-of-the-art text embedder among open models for Uzbek. We also demonstrate that closing the performance gap via fine-tuning is both impractical due to the intensive hardware demands of long-context legal Q&A and unnecessary, given that legal acts change frequently. We support this by reporting a negative result from a QLoRA experiment. We distill practical guidance for similar deployments, drawn from a system serving real users in production. We release our benchmarks, evaluation code and the fine-tuned embedder (UTE-1) at this https URL to support future work on low-resource legal NLP.