Continuous Decomposition of Granularity for Neural Paraphrase Generation

Xiaodong Gu, Zhaowei Zhang, Sang-Woo Lee, Kang Min Yoo, Jung-Woo Ha


Abstract
While Transformers have had significant success in paragraph generation, they treat sentences as linear sequences of tokens and often neglect their hierarchical information. Prior work has shown that decomposing the levels of granularity (e.g., word, phrase, or sentence) for input tokens has produced substantial improvements, suggesting the possibility of enhancing Transformers via more fine-grained modeling of granularity. In this work, we present continuous decomposition of granularity for neural paraphrase generation (C-DNPG): an advanced extension of multi-head self-attention with: 1) a granularity head that automatically infers the hierarchical structure of a sentence by neurally estimating the granularity level of each input token; and 2) two novel attention masks, namely, granularity resonance and granularity scope, to efficiently encode granularity into attention. Experiments on two benchmarks, including Quora question pairs and Twitter URLs have shown that C-DNPG outperforms baseline models by a significant margin. Qualitative analysis reveals that C-DNPG indeed captures fine-grained levels of granularity with effectiveness.
Anthology ID:
2022.coling-1.554
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
6369–6378
Language:
URL:
https://aclanthology.org/2022.coling-1.554
DOI:
Bibkey:
Cite (ACL):
Xiaodong Gu, Zhaowei Zhang, Sang-Woo Lee, Kang Min Yoo, and Jung-Woo Ha. 2022. Continuous Decomposition of Granularity for Neural Paraphrase Generation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6369–6378, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Continuous Decomposition of Granularity for Neural Paraphrase Generation (Gu et al., COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.554.pdf
Code
 guxd/c-dnpg