Bhawani Shankar Leelar

Author directory

2026

Production text-to-code systems increasingly use LLMs to translate natural-language analytics requests into executable SQL and pandas code, serving both business users, who describe desired results in application terms, and technical users, who name tables and columns directly. Existing Text-to-SQL and Text-to-Python benchmarks provide coverage, but they do not isolate when SQL and pandas performance diverge. We introduce BIRD-Persona, a persona-stratified extension of BIRD-Python that adds novice/expert rewrites (business-style and technical-style phrasing) to tasks with matched SQL and pandas ground truth, plus schema variants and task-structure annotations. Across 1,534 tasks, nine models, and 245,592 evaluations, BIRD-Persona reproduces the previously reported SQL-over-Python gap: SQL outperforms pandas for eight of nine systems. The gap is not uniform: expert, schema-explicit prompts improve both paradigms but help SQL more, schema representations have mixed model-dependent effects, and selected-column burden widens the gap while subqueries do not. By making user phrasing a controlled benchmark dimension, BIRD-Persona provides a foundation for persona-aware and paradigm-aware evaluation of natural-language analytics systems.