Bias Analysis and Mitigation in the Evaluation of Authorship Verification

Janek Bevendorff, Matthias Hagen, Benno Stein, Martin Potthast


Abstract
The PAN series of shared tasks is well known for its continuous and high quality research in the field of digital text forensics. Among others, PAN contributions include original corpora, tailored benchmarks, and standardized experimentation platforms. In this paper we review, theoretically and practically, the authorship verification task and conclude that the underlying experiment design cannot guarantee pushing forward the state of the art—in fact, it allows for top benchmarking with a surprisingly straightforward approach. In this regard, we present a “Basic and Fairly Flawed” (BAFF) authorship verifier that is on a par with the best approaches submitted so far, and that illustrates sources of bias that should be eliminated. We pinpoint these sources in the evaluation chain and present a refined authorship corpus as effective countermeasure.
Anthology ID:
P19-1634
Volume:
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2019
Address:
Florence, Italy
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6301–6306
Language:
URL:
https://aclanthology.org/P19-1634
DOI:
10.18653/v1/P19-1634
Bibkey:
Cite (ACL):
Janek Bevendorff, Matthias Hagen, Benno Stein, and Martin Potthast. 2019. Bias Analysis and Mitigation in the Evaluation of Authorship Verification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6301–6306, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
Bias Analysis and Mitigation in the Evaluation of Authorship Verification (Bevendorff et al., ACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/P19-1634.pdf
Code
 webis-de/acl-19