AI coding assistants powered by large language models (LLMs) have transformed software development, significantly boosting productivity. While existing benchmarks evaluate the correctness and security of LLM-generated code, they are limited to single-turn tasks that cannot capture the conversational nature of real-world development, where requirements evolve across turns and tools increasingly generate incremental diffs rather than complete programs. We introduce MT-Sec, the first benchmark to jointly evaluate correctness and security in multi-turn coding scenarios. MT-Sec covers four interaction types (expansion, editing, refactoring, and debugging) reflecting common development workflows. We construct multi-turn tasks using human-curated instructions and feedback-driven interactions grounded in dynamic test suites. Evaluating 31 LLMs and 3 agent scaffolds, we find that models struggle most with expansion and editing, but improve significantly in debugging when provided with feedback. Agent-based systems often underperform their base models in automated multi-turn settings due to brittle tool use. We also evaluate multi-turn code-diff generation, an unexplored but practical setting, and observe higher failure rates compared to full-program generation. Our findings underscore the need for multi-turn benchmarks to drive safer AI-assisted development.

Recently, ChatGPT has demonstrated remarkable performance in various downstream tasks such as open-domain question answering, machine translation, and code generation. As a general-purpose task solver, an intriguing inquiry arises: Does ChatGPT itself know that it does not know, without any access to internal states? In response to this query, we present an initial evaluation of ChatGPT for black-box calibration. We designed three types of proxy confidence, from three perspectives to assess its performance. Experiments are conducted on five datasets, spanning four tasks, and the results show that ChatGPT has a degree of capability for black-box calibration. Specifically, proxy confidence displayed a significantly positive Pearson correlation (95.16%) with accuracy in the TruthfulQA dataset, while revealing a negative correlation in the ModAr dataset. We delved deeper into ChatGPT’s black-box calibration ability by examining failure cases in the ModAr dataset. Our analysis revealed that ChatGPT’s tendency to exhibit overconfidence may stem from its reliance on semantic priors. Furthermore, we investigated why ChatGPT performs relatively well in TruthfulQA. The findings suggest that ChatGPT might implicitly acquire calibration skills during the reinforcement learning process, rather than relying solely on simplistic heuristics.