Production text-to-code systems increasingly use LLMs to translate natural-language analytics requests into executable SQL and pandas code, serving both business users, who describe desired results in application terms, and technical users, who name tables and columns directly. Existing Text-to-SQL and Text-to-Python benchmarks provide coverage, but they do not isolate when SQL and pandas performance diverge. We introduce BIRD-Persona, a persona-stratified extension of BIRD-Python that adds novice/expert rewrites (business-style and technical-style phrasing) to tasks with matched SQL and pandas ground truth, plus schema variants and task-structure annotations. Across 1,534 tasks, nine models, and 245,592 evaluations, BIRD-Persona reproduces the previously reported SQL-over-Python gap: SQL outperforms pandas for eight of nine systems. The gap is not uniform: expert, schema-explicit prompts improve both paradigms but help SQL more, schema representations have mixed model-dependent effects, and selected-column burden widens the gap while subqueries do not. By making user phrasing a controlled benchmark dimension, BIRD-Persona provides a foundation for persona-aware and paradigm-aware evaluation of natural-language analytics systems.

In task-oriented conversational AI evaluation, unsupervised methods poorly correlate with human judgments, and supervised approaches lack generalization. Recent advances in large language models (LLMs) show robust zero- and few-shot capabilities across NLP tasks. Our paper explores using LLMs for automated dialogue quality evaluation, experimenting with various configurations on public and proprietary datasets. Manipulating factors such as model size, in-context examples, and selection techniques, we examine “chain-of-thought” (CoT) reasoning and label extraction procedures. Our results show that (1) larger models yield more accurate dialogue labels; (2) algorithmic selection of in-context examples outperforms random selection,; (3) CoT reasoning where an LLM is asked to provide justifications before outputting final labels improves performance; and (4) fine-tuned LLMs outperform out-of-the-box ones. In addition, we find that suitably tuned LLMs exhibit high accuracy in dialogue evaluation compared to human judgments.

Large language models can accumulate incorrect or outdated knowledge as the real world evolves. Compared to typical solutions such as retraining, retrieval augmented generation, model editing offers an effective yet low cost solution to address this issue. However, existing model editing algorithms employ manual selection of edit layers, which requires prior domain knowledge or expensive architecture-specific empirical layer selection methods, such as causal tracing. In this work, we propose SaLEM (Salient Layers Editing Model), an efficient solution for data driven layer selection for the model editing task. Our solution utilizes layer-wise saliency maps for layer selection, and matches the accuracy of prior approaches but with only 1/3 of their edits, enabling efficient updates to the parametric knowledge in large language models.