Phuvis Kerdpramote
Author directory2026
Composing Within Constraints: A Benchmark for Thai Classical Poetry in Large Language Models
Nutchanon Yongsatianchot | Phuvis Kerdpramote
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)
Nutchanon Yongsatianchot | Phuvis Kerdpramote
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)
Thai is spoken by tens of millions, yet its literary tradition is largely absent from LLM evaluation. We present ThaiProsody, a benchmark for composing Thai classical poetry. Composing classical Thai verse means choosing words that jointly satisfy several hard formal constraints (fixed syllable counts, written tone-mark positions, heavy/light syllable patterns, and an interlocking web of rhymes) while staying on a given topic. ThaiProsody crosses four canonical verse forms with 40 topics and grades each poem with an open, deterministic rule checker for tone, weight, and rhyme, a syllable counter validated against hand-segmented gold, and a separate content judge. Across 17 models in 24 configurations, the task is far from solved. The best model passes 94.7% of rule checks, yet one stanza in four still breaks a rule; the best open-weight model writes a fully valid stanza only a quarter of the time, and no Thai-trained model more than once in 160 tries. Rhyme is the most frequently broken constraint, and khlong and chan, the forms that add tone or weight constraints, are the hardest. Thai-trained 8B models stay close to their general base model, while a general 31B model scores more than three times as high as a Thai 30B model.