Exact Paired-Permutation Testing for Structured Test Statistics
Ran
Zmigrod
author
Tim
Vieira
author
Ryan
Cotterell
author
2022-07
text
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Association for Computational Linguistics
Seattle, United States
conference publication
Significance testing—especially the paired-permutation test—has played a vital role in developing NLP systems to provide confidence that the difference in performance between two systems (i.e., the test statistic) is not due to luck. However, practitioners rely on Monte Carlo approximation to perform this test due to a lack of a suitable exact algorithm. In this paper, we provide an efficient exact algorithm for the paired-permutation test for a family of structured test statistics. Our algorithm runs in \mathcalO(G N (łog GN )(łog N)) time where N is the dataset size and G is the range of the test statistic. We found that our exact algorithm was 10x faster than the Monte Carlo approximation with 20000 samples on a common dataset
zmigrod-etal-2022-exact
10.18653/v1/2022.naacl-main.360
https://aclanthology.org/2022.naacl-main.360
2022-07
4894
4902