Jinjun Cai
Author directory2026
ATC-Bench: A Simulation-Grounded Benchmark for Evaluating LLM-Generated Air Traffic Control Instructions
Jinjun Cai | Jiaxing Li | Xingyue Wang | Weihuang Zheng | Youyong Kong | Hui Ding
Findings of the Association for Computational Linguistics: EMNLP 2026
Jinjun Cai | Jiaxing Li | Xingyue Wang | Weihuang Zheng | Youyong Kong | Hui Ding
Findings of the Association for Computational Linguistics: EMNLP 2026
Air traffic control (ATC) requires structured, safety-critical decision making whose effects unfold through aircraft trajectories, so text-level correctness alone is insufficient for evaluating large language models (LLMs) in this domain. We introduce ATC-Bench, a simulation-grounded benchmark for ATC instruction generation with Chinese-facing scenario descriptions and controller-style instruction fields. Given a structured Chinese traffic situation, a model generates a JSON control plan that is translated into BlueSky commands and evaluated through closed-loop simulation against separation, no-fly-zone (NFZ) avoidance, and route-recovery constraints. The benchmark spans four task families that form a controlled complexity gradient: single-aircraft NFZ avoidance, pairwise conflict resolution, joint conflict-airspace constraints, and multi-aircraft conflict graphs. Experiments with 11 representative LLM endpoints show that while most models produce format-valid instructions, task success varies widely across models and task families: the highest-scoring endpoint under our fixed protocol achieves 63.5% macro-average task success rate (TSR), while the two lowest endpoints fall below 25%, with the largest gaps appearing under joint constraints and multi-aircraft coordination. These results expose a persistent execution gap between static validity and simulation-passing behavior, motivating simulation-in-the-loop evaluation for LLM-based decision making in safety-critical aviation settings.