Although large language models (LLMs) perform much worse than traditional supervised models on the semantic role labeling (SRL) task, our closer investigation reveals two important findings. First, the existing labeled SRL data is substantially flawed, i.e., of unsatisfactory quality. Second, LLMs can serve as an effective tool for detecting annotation errors and deciding which labels require human review, from the perspective of minimizing human effort. This paper proposes an LLM-assisted data quality improvement approach, and applies it to two SRL evaluation datasets from two different domains. With our approach, human annotators need to review only 11% of the originally annotated arguments flagged for manual inspection, among which 40% are truly erroneous. Meanwhile, our approach suggests 20% new arguments potentially missed in the original labels for human review, among which 60% are indeed missing. In total, our approach locates and corrects 21% errors in the two evaluation datasets. Experiments show that both traditional supervised models and LLMs perform much better on the revised evaluation datasets.

During the past decade, neural network models have made tremendous progress on in-domain semantic role labeling (SRL). However, performance drops dramatically under the out-of-domain setting. In order to facilitate research on cross-domain SRL, this paper presents MuCPAD, a multi-domain Chinese predicate-argument dataset, which consists of 30,897 sentences and 92,051 predicates from six different domains. MuCPAD exhibits three important features. 1) Based on a frame-free annotation methodology, we avoid writing complex frames for new predicates. 2) We explicitly annotate omitted core arguments to recover more complete semantic structure, considering that omission of content words is ubiquitous in multi-domain Chinese texts. 3) We compile 53 pages of annotation guidelines and adopt strict double annotation for improving data quality. This paper describes in detail the annotation methodology and annotation process of MuCPAD, and presents in-depth data analysis. We also give benchmark results on cross-domain SRL based on MuCPAD.