Serena Lutong Wang
Author directory2026
A Tutorial on Measurement Science for Language Model Evaluation
Sang T. Truong | Serena Lutong Wang | Rodolfo Corona | Sanmi Koyejo
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: Tutorial Abstracts
Sang T. Truong | Serena Lutong Wang | Rodolfo Corona | Sanmi Koyejo
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: Tutorial Abstracts
As language models are increasingly optimized against the benchmarks used to evaluate them and deployed across diverse global populations, the assumptions behind evaluation are quietly breaking down. This tutorial focuses on a scientific approach for language model evaluation: a measurement-theoretic framework that treats LLM evaluation as a scientific measurement problem. We organize three hours around the classical structure of a measurement instrument and give NLP researchers usable tools for each part: (1) validity–when a benchmark score actually measures the capability it claims to, with emphasis on cross-cultural and cross-lingual transfer; (2) predictive measurement–inferring capability efficiently from sparse data via latent-variable models and cold-start prediction; and (3) design and incentives–why some benchmarks are gameable and what mechanisms improve robustness under explicit assumptions about evaluator access, developer objectives, and information leakage. The tutorial connects language model evaluation, latent-variable modeling, and mechanism design, three literatures that NLP researchers rarely encounter together, and equips the audience to build and interpret evaluations that remain trustworthy under deployment and strategic pressure.