Dayong Yang

Author directory

2026

Modern coding agents are increasingly deployed for realistic software engineering, yet existing benchmarks primarily evaluate static, fixed-target tasks, failing to capture the long-horizon, multi-turn nature of real-world development. We introduce MACBench, a novel benchmark evaluating agents on sustained frontend web application development where text-based user requirements continuously evolve within a persistent workspace. Comprising 28 repository instances across 8 domains, MACBench features 699 dialogue turns and 956 evaluation items. It explicitly models realistic interaction patterns, challenging agents to maintain cross-turn coherence and continuous executability. Our dual-branch evaluation framework assesses both process robustness and final artifact quality, uniquely integrating intermediate viability checks with bounded runtime-feedback repair. Benchmarking 4 contemporary models reveals a severe reliability gap: the best-performing agent achieves only a 70.6% checklist success rate and a 10.7% strict instance success rate, struggling to satisfy requirements accumulated over long sessions. Furthermore, ablation studies reveal diminishing and sometimes negative returns from runtime-feedback repair: while it improves checklist success by an average of 10.4% and aids immediate failure recovery, excessive repair attempts frequently degrade final functional outcomes, suggesting they may introduce regressions or side effects in the evolving codebase. Finally, failure analysis identifies complex cross-turn application logic as the dominant bottleneck, accounting for 33–50% of unresolved errors.