The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task

Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, Arpit Mittal


Abstract
The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) shared task, asks participating systems to determine whether human-authored claims are Supported or Refuted based on evidence retrieved from Wikipedia (or NotEnoughInfo if the claim cannot be verified). Compared to the FEVER 2018 shared task, the main challenge is the addition of structured data (tables and lists) as a source of evidence. The claims in the FEVEROUS dataset can be verified using only structured evidence, only unstructured evidence, or a mixture of both. Submissions are evaluated using the FEVEROUS score that combines label accuracy and evidence retrieval. Unlike FEVER 2018, FEVEROUS requires partial evidence to be returned for NotEnoughInfo claims, and the claims are longer and thus more complex. The shared task received 13 entries, six of which were able to beat the baseline system. The winning team was “Bust a move!”, achieving a FEVEROUS score of 27% (+9% compared to the baseline). In this paper we describe the shared task, present the full results and highlight commonalities and innovations among the participating systems.
Anthology ID:
2021.fever-1.1
Volume:
Proceedings of the Fourth Workshop on Fact Extraction and VERification (FEVER)
Month:
November
Year:
2021
Address:
Dominican Republic
Venue:
FEVER
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1–13
Language:
URL:
https://aclanthology.org/2021.fever-1.1
DOI:
10.18653/v1/2021.fever-1.1
Bibkey:
Cite (ACL):
Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task. In Proceedings of the Fourth Workshop on Fact Extraction and VERification (FEVER), pages 1–13, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
The Fact Extraction and VERification Over Unstructured and Structured information (FEVEROUS) Shared Task (Aly et al., FEVER 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.fever-1.1.pdf
Video:
 https://aclanthology.org/2021.fever-1.1.mp4
Data
FEVERFEVEROUS