Gated Transformer for Robust De-noised Sequence-to-Sequence Modelling

Ayan Sengupta, Amit Kumar, Sourabh Kumar Bhattacharjee, Suman Roy


Abstract
Robust sequence-to-sequence modelling is an essential task in the real world where the inputs are often noisy. Both user-generated and machine generated inputs contain various kinds of noises in the form of spelling mistakes, grammatical errors, character recognition errors, all of which impact downstream tasks and affect interpretability of texts. In this work, we devise a novel sequence-to-sequence architecture for detecting and correcting different real world and artificial noises (adversarial attacks) from English texts. Towards that we propose a modified Transformer-based encoder-decoder architecture that uses a gating mechanism to detect types of corrections required and accordingly corrects texts. Experimental results show that our gated architecture with pre-trained language models perform significantly better that the non-gated counterparts and other state-of-the-art error correction models in correcting spelling and grammatical errors. Extrinsic evaluation of our model on Machine Translation (MT) and Summarization tasks show the competitive performance of the model against other generative sequence-to-sequence models under noisy inputs.
Anthology ID:
2021.findings-emnlp.309
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2021
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
3645–3657
Language:
URL:
https://aclanthology.org/2021.findings-emnlp.309
DOI:
10.18653/v1/2021.findings-emnlp.309
Bibkey:
Cite (ACL):
Ayan Sengupta, Amit Kumar, Sourabh Kumar Bhattacharjee, and Suman Roy. 2021. Gated Transformer for Robust De-noised Sequence-to-Sequence Modelling. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3645–3657, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Gated Transformer for Robust De-noised Sequence-to-Sequence Modelling (Sengupta et al., Findings 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.findings-emnlp.309.pdf
Software:
 2021.findings-emnlp.309.Software.zip
Video:
 https://aclanthology.org/2021.findings-emnlp.309.mp4
Code
 victor7246/gated-transformer
Data
IMDb Movie ReviewsWMT 2014