Wanyi Chen
Author directory2026
Attributing Structured-Output Gains in Function Calling: Interface Alignment versus Procedural Transfer
Wanyi Chen | Daoyuan Chen | Fang Kong
Findings of the Association for Computational Linguistics: EMNLP 2026
Wanyi Chen | Daoyuan Chen | Fang Kong
Findings of the Association for Computational Linguistics: EMNLP 2026
Function-calling benchmarks assess whether language models can select appropriate tools and supply the required arguments. Because their scores also depend on expressing calls in an evaluator-specific format, gains from prompt-prepended skills may reflect interface compliance rather than transferable task-solving procedures. We introduce a four-layer gain-attribution protocol for prompt-prepended skill injection, combining canonicalized rescoring, format-only controls, repaired/balanced induction, and portability checks. Applied to the Berkeley Function Calling Leaderboard (BFCL) and scoped with API-Bank, MATH-500, and MultiHop-RAG, the protocol shows that several apparent gains are better attributed to interface alignment than to procedural transfer: format-only prompts match or exceed full skills in key BFCL cells, repaired/balanced induction removes the largest sub-frontier gains, and API-Bank target-native gains are matched within 0.5 percentage points (pp) by length-matched generic procedural prompts. These findings treat format compliance as a useful engineering capability while clarifying what a structured-output score certifies. We release BFCL-CANONICAL and recommend canonicalized metrics, balanced induction, and format-only baselines for function-calling skill-gain attribution. Code is available at https://github.com/wanyichen06/skill-injection-attribution.
EVM-QuestBench: An Execution-Grounded Benchmark for Natural-Language Transaction Code Generation
Pei Yang | Wanyi Chen | Ke Wang | Lynn Ai | Eric Yang | Tianyu Shi
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Pei Yang | Wanyi Chen | Ke Wang | Lynn Ai | Eric Yang | Tianyu Shi
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Large language models are increasingly applied to various development scenarios. However, in on-chain transaction scenarios, even a minor error can cause irreversible loss for users. Existing evaluations often overlook execution accuracy and safety. We introduce EVM-QuestBench, an execution-grounded benchmark for natural-language transaction-script generation on EVM-compatible chains. The benchmark employs dynamic evaluation: instructions are sampled from template pools, numeric parameters are drawn from predefined intervals, and validators verify outcomes against these instantiated values. EVM-QuestBench contains 107 tasks (62 atomic, 45 composite). Its modular architecture enables rapid task development. The runner executes scripts on a forked EVM chain with snapshot isolation; composite tasks apply step-efficiency decay. We evaluate 20 models with 5 independent rounds each and find large performance gaps, with split scores revealing persistent asymmetry between single-action precision and multi-step workflow completion. Code: https://github.com/OpenEdgeHQ/EVM-quest-bench.