Human-in-the-Loop Synthetic Text Data Inspection with Provenance Tracking

Hong Jin Kang, Fabrice Harel-Canada, Muhammad Ali Gulzar, Nanyun Peng, Miryung Kim


Abstract
Data augmentation techniques apply transformations to existing texts to generate additional data. The transformations may produce low-quality texts, where the meaning of the text is changed and the text may even be mangled beyond human comprehension. Analyzing the synthetically generated texts and their corresponding labels is slow and demanding. To winnow out texts with incorrect labels, we develop INSPECTOR, a human-in-the-loop data inspection technique. INSPECTOR combines the strengths of provenance tracking techniques with assistive labeling. INSPECTOR allows users to group related texts by their transformation provenance, i.e., the transformations applied to the original text, or feature provenance, the linguistic features of the original text. For assistive labeling, INSPECTOR computes metrics that approximate data quality, and allows users to compare the corresponding label of each text against the predictions of a large language model. In a user study, INSPECTOR increases the number of texts with correct labels identified by on a sentiment analysis task and by on a hate speech detection task. The participants found grouping the synthetically generated texts by their common transformation to be the most useful technique. Surprisingly, grouping texts by common linguistic features was perceived to be unhelpful. Contrary to prior work, our study finds that no single technique obviates the need for human inspection effort. This validates the design of INSPECTOR which combines both analysis of data provenance and assistive labeling to reduce human inspection effort.
Anthology ID:
2024.findings-naacl.197
Volume:
Findings of the Association for Computational Linguistics: NAACL 2024
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3118–3129
Language:
URL:
https://aclanthology.org/2024.findings-naacl.197
DOI:
Bibkey:
Cite (ACL):
Hong Jin Kang, Fabrice Harel-Canada, Muhammad Ali Gulzar, Nanyun Peng, and Miryung Kim. 2024. Human-in-the-Loop Synthetic Text Data Inspection with Provenance Tracking. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3118–3129, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Human-in-the-Loop Synthetic Text Data Inspection with Provenance Tracking (Kang et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-naacl.197.pdf
Copyright:
 2024.findings-naacl.197.copyright.pdf