AI coding assistants powered by large language models (LLMs) have transformed software development, significantly boosting productivity. While existing benchmarks evaluate the correctness and security of LLM-generated code, they are limited to single-turn tasks that cannot capture the conversational nature of real-world development, where requirements evolve across turns and tools increasingly generate incremental diffs rather than complete programs. We introduce MT-Sec, the first benchmark to jointly evaluate correctness and security in multi-turn coding scenarios. MT-Sec covers four interaction types (expansion, editing, refactoring, and debugging) reflecting common development workflows. We construct multi-turn tasks using human-curated instructions and feedback-driven interactions grounded in dynamic test suites. Evaluating 31 LLMs and 3 agent scaffolds, we find that models struggle most with expansion and editing, but improve significantly in debugging when provided with feedback. Agent-based systems often underperform their base models in automated multi-turn settings due to brittle tool use. We also evaluate multi-turn code-diff generation, an unexplored but practical setting, and observe higher failure rates compared to full-program generation. Our findings underscore the need for multi-turn benchmarks to drive safer AI-assisted development.

The rapid growth in natural language processing (NLP) research has led to numerous new models, outpacing our understanding of how they compare to established ones. One major reason for this difficulty is saturating benchmarks, which may not well reflect differences in model performance in the wild. In this work, we introduce a novel framework to compare two NLP models by revealing their shared invariance to interpretable input perturbations targeting a specific linguistic capability. Via experiments on models from the same and different architecture families, this framework offers insights about how changes in models (e.g., distillation, size increase) affect linguistic capabilities. Furthermore, our framework enables evaluation of invariances between commercial black-box models (e.g., InstructGPT family) and models that are better understood (e.g., GPT-2). Across experiments, we observe that large language models share many invariances encoded by models of various sizes, whereas the invariances by large models are only shared by other large models. Possessing a wide variety of invariances may be key to the recent successes of large language models, and our framework can shed light on the types of invariances retained or emerging in new models. We make the code publicly available.