Paired Examples as Indirect Supervision in Latent Decision Models

Nitish Gupta, Sameer Singh, Matt Gardner, Dan Roth


Abstract
Compositional, structured models are appealing because they explicitly decompose problems and provide interpretable intermediate outputs that give confidence that the model is not simply latching onto data artifacts. Learning these models is challenging, however, because end-task supervision only provides a weak indirect signal on what values the latent decisions should take. This often results in the model failing to learn to perform the intermediate tasks correctly. In this work, we introduce a way to leverage paired examples that provide stronger cues for learning latent decisions. When two related training examples share internal substructure, we add an additional training objective to encourage consistency between their latent decisions. Such an objective does not require external supervision for the values of the latent output, or even the end task, yet provides an additional training signal to that provided by individual training examples themselves. We apply our method to improve compositional question answering using neural module networks on the DROP dataset. We explore three ways to acquire paired questions in DROP: (a) discovering naturally occurring paired examples within the dataset, (b) constructing paired examples using templates, and (c) generating paired examples using a question generation model. We empirically demonstrate that our proposed approach improves both in- and out-of-distribution generalization and leads to correct latent decision predictions.
Anthology ID:
2021.emnlp-main.466
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5774–5785
Language:
URL:
https://aclanthology.org/2021.emnlp-main.466
DOI:
10.18653/v1/2021.emnlp-main.466
Bibkey:
Cite (ACL):
Nitish Gupta, Sameer Singh, Matt Gardner, and Dan Roth. 2021. Paired Examples as Indirect Supervision in Latent Decision Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5774–5785, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Paired Examples as Indirect Supervision in Latent Decision Models (Gupta et al., EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.466.pdf
Data
DROP