Cross-Policy Compliance Detection via Question Answering

Marzieh Saeidi, Majid Yazdani, Andreas Vlachos


Abstract
Policy compliance detection is the task of ensuring that a scenario conforms to a policy (e.g. a claim is valid according to government rules or a post in an online platform conforms to community guidelines). This task has been previously instantiated as a form of textual entailment, which results in poor accuracy due to the complexity of the policies. In this paper we propose to address policy compliance detection via decomposing it into question answering, where questions check whether the conditions stated in the policy apply to the scenario, and an expression tree combines the answers to obtain the label. Despite the initial upfront annotation cost, we demonstrate that this approach results in better accuracy, especially in the cross-policy setup where the policies during testing are unseen in training. In addition, it allows us to use existing question answering models pre-trained on existing large datasets. Finally, it explicitly identifies the information missing from a scenario in case policy compliance cannot be determined. We conduct our experiments using a recent dataset consisting of government policies, which we augment with expert annotations and find that the cost of annotating question answering decomposition is largely offset by improved inter-annotator agreement and speed.
Anthology ID:
2021.emnlp-main.678
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8622–8632
Language:
URL:
https://aclanthology.org/2021.emnlp-main.678
DOI:
10.18653/v1/2021.emnlp-main.678
Bibkey:
Cite (ACL):
Marzieh Saeidi, Majid Yazdani, and Andreas Vlachos. 2021. Cross-Policy Compliance Detection via Question Answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8622–8632, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Cross-Policy Compliance Detection via Question Answering (Saeidi et al., EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.678.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.678.mp4
Data
BoolQShARC