IDANI: Inference-time Domain Adaptation via Neuron-level Interventions

Omer Antverg, Eyal Ben-David, Yonatan Belinkov


Abstract
Large pre-trained models are usually fine-tuned on downstream task data, and tested on unseen data. When the train and test data come from different domains, the model is likely to struggle, as it is not adapted to the test domain. We propose a new approach for domain adaptation (DA), using neuron-level interventions: We modify the representation of each test example in specific neurons, resulting in a counterfactual example from the source domain, which the model is more familiar with. The modified example is then fed back into the model. While most other DA methods are applied during training time, ours is applied during inference only, making it more efficient and applicable. Our experiments show that our method improves performance on unseen domains.
Anthology ID:
2022.deeplo-1.3
Volume:
Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing
Month:
July
Year:
2022
Address:
Hybrid
Editors:
Colin Cherry, Angela Fan, George Foster, Gholamreza (Reza) Haffari, Shahram Khadivi, Nanyun (Violet) Peng, Xiang Ren, Ehsan Shareghi, Swabha Swayamdipta
Venue:
DeepLo
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
21–29
Language:
URL:
https://aclanthology.org/2022.deeplo-1.3
DOI:
10.18653/v1/2022.deeplo-1.3
Bibkey:
Cite (ACL):
Omer Antverg, Eyal Ben-David, and Yonatan Belinkov. 2022. IDANI: Inference-time Domain Adaptation via Neuron-level Interventions. In Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing, pages 21–29, Hybrid. Association for Computational Linguistics.
Cite (Informal):
IDANI: Inference-time Domain Adaptation via Neuron-level Interventions (Antverg et al., DeepLo 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.deeplo-1.3.pdf
Video:
 https://aclanthology.org/2022.deeplo-1.3.mp4
Code
 technion-cs-nlp/idani