Bhawani Shankar Leelar
Author directory2026
BIRD-Persona: A Persona-Aware Benchmark for SQL and pandas Analytics Code Generation
Davin Hill | Pawan Chorasiya | Bhawani Shankar Leelar | Oren Avram | Robert E. Tillman | Tamer Soliman
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Davin Hill | Pawan Chorasiya | Bhawani Shankar Leelar | Oren Avram | Robert E. Tillman | Tamer Soliman
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Production text-to-code systems increasingly use LLMs to translate natural-language analytics requests into executable SQL and pandas code, serving both business users, who describe desired results in application terms, and technical users, who name tables and columns directly. Existing Text-to-SQL and Text-to-Python benchmarks provide coverage, but they do not isolate when SQL and pandas performance diverge. We introduce BIRD-Persona, a persona-stratified extension of BIRD-Python that adds novice/expert rewrites (business-style and technical-style phrasing) to tasks with matched SQL and pandas ground truth, plus schema variants and task-structure annotations. Across 1,534 tasks, nine models, and 245,592 evaluations, BIRD-Persona reproduces the previously reported SQL-over-Python gap: SQL outperforms pandas for eight of nine systems. The gap is not uniform: expert, schema-explicit prompts improve both paradigms but help SQL more, schema representations have mixed model-dependent effects, and selected-column burden widens the gap while subqueries do not. By making user phrasing a controlled benchmark dimension, BIRD-Persona provides a foundation for persona-aware and paradigm-aware evaluation of natural-language analytics systems.