GPoeT: a Language Model Trained for Rhyme Generation on Synthetic Data

Andrei Popescu-Belis, Àlex R. Atrio, Bastien Bernath, Etienne Boisson, Teo Ferrari, Xavier Theimer-Lienhard, Giorgos Vernikos


Abstract
Poem generation with language models requires the modeling of rhyming patterns. We propose a novel solution for learning to rhyme, based on synthetic data generated with a rule-based rhyming algorithm. The algorithm and an evaluation metric use a phonetic dictionary and the definitions of perfect and assonant rhymes. We fine-tune a GPT-2 English model with 124M parameters on 142 MB of natural poems and find that this model generates consecutive rhymes infrequently (11%). We then fine-tune the model on 6 MB of synthetic quatrains with consecutive rhymes (AABB) and obtain nearly 60% of rhyming lines in samples generated by the model. Alternating rhymes (ABAB) are more difficult to model because of longer-range dependencies, but they are still learnable from synthetic data, reaching 45% of rhyming lines in generated samples.
Anthology ID:
2023.latechclfl-1.2
Volume:
Proceedings of the 7th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature
Month:
May
Year:
2023
Address:
Dubrovnik, Croatia
Editors:
Stefania Degaetano-Ortlieb, Anna Kazantseva, Nils Reiter, Stan Szpakowicz
Venue:
LaTeCHCLfL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
10–20
Language:
URL:
https://aclanthology.org/2023.latechclfl-1.2
DOI:
10.18653/v1/2023.latechclfl-1.2
Bibkey:
Cite (ACL):
Andrei Popescu-Belis, Àlex R. Atrio, Bastien Bernath, Etienne Boisson, Teo Ferrari, Xavier Theimer-Lienhard, and Giorgos Vernikos. 2023. GPoeT: a Language Model Trained for Rhyme Generation on Synthetic Data. In Proceedings of the 7th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 10–20, Dubrovnik, Croatia. Association for Computational Linguistics.
Cite (Informal):
GPoeT: a Language Model Trained for Rhyme Generation on Synthetic Data (Popescu-Belis et al., LaTeCHCLfL 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.latechclfl-1.2.pdf
Video:
 https://aclanthology.org/2023.latechclfl-1.2.mp4