Songhua Liu

Author directory

2026

Multi-turn tool calling is a core evaluation scenario for large language model (LLM) agents. On public tool-calling benchmarks, open-weight models now approach or even surpass closed-source frontier models in aggregate accuracy. However, aggregate accuracy averages over many different multi-turn situations and obscures whether progress is balanced across them, masking failure redistribution. We propose an action-class-oriented diagnostic framework that decomposes failures into two orthogonal dimensions: capability and calibration. The framework operates over a four-class action space (tool_call/ask/refuse/confirm) and introduces a self-revealing upper bound, Acc ≤ GAR (Gold Action Recall), whose two violation directions separately localize action-class miscalibration and sub-action failures. We validate it on an extensive panel of tool-calling models that includes both open-weight families and closed-source frontier systems, across multiple multi-turn tool-calling benchmarks for cross-benchmark robustness. Across our panel the bottleneck is calibration, not capability: every model can produce all four classes, but which one it fires in which situation is highly model-specific. On no-tool-call scenarios, a top-ranked family’s GAR is roughly one-quarter that of size-matched peers, a 64.5 percentage point (pp) gap, indicating its ranking reflects benchmark-distribution fitting rather than genuine in-context calibration. We further show the calibration profile can be reshaped through context manipulation alone, with the reshaping direction strongly model-specific. We recommend that multi-turn tool-calling evaluations report per-category (GAR, tool-call-emit rate, Acc) triples, exposing the calibration structure aggregate accuracy hides. Our framework provides a falsifiable, reproducible diagnostic foundation for evaluation protocol design, training objective alignment, and inference-time calibration research.

2025

Recent years have witnessed rapid advancements in Large Language Models (LLMs). Nevertheless, it remains unclear whether state-of-the-art LLMs can infer the author of an anonymous research paper solely from the text, without any additional information. To investigate this novel challenge, which we define as Open-World Authorship Attribution, we introduce a benchmark comprising thousands of research papers across various fields to quantitatively assess model capabilities. Then, at the core of this paper, we tailor a two-stage framework to tackle this problem: candidate selection and authorship decision. Specifically, in the first stage, LLMs are prompted to generate multi-level key information, which are then used to identify potential candidates through Internet searches. In the second stage, we introduce key perspectives to guide LLMs in determining the most likely author from these candidates. Extensive experiments on our benchmark demonstrate the effectiveness of the proposed approach, achieving 60.7% and 44.3% accuracy in the two stages, respectively. We will release our benchmark and source codes to facilitate future research in this field.