YASO: A Targeted Sentiment Analysis Evaluation Dataset for Open-Domain Reviews

Matan Orbach, Orith Toledo-Ronen, Artem Spector, Ranit Aharonov, Yoav Katz, Noam Slonim


Abstract
Current TSA evaluation in a cross-domain setup is restricted to the small set of review domains available in existing datasets. Such an evaluation is limited, and may not reflect true performance on sites like Amazon or Yelp that host diverse reviews from many domains. To address this gap, we present YASO – a new TSA evaluation dataset of open-domain user reviews. YASO contains 2,215 English sentences from dozens of review domains, annotated with target terms and their sentiment. Our analysis verifies the reliability of these annotations, and explores the characteristics of the collected data. Benchmark results using five contemporary TSA systems show there is ample room for improvement on this challenging new dataset. YASO is available at https://github.com/IBM/yaso-tsa.
Anthology ID:
2021.emnlp-main.721
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
9154–9173
Language:
URL:
https://aclanthology.org/2021.emnlp-main.721
DOI:
10.18653/v1/2021.emnlp-main.721
Bibkey:
Cite (ACL):
Matan Orbach, Orith Toledo-Ronen, Artem Spector, Ranit Aharonov, Yoav Katz, and Noam Slonim. 2021. YASO: A Targeted Sentiment Analysis Evaluation Dataset for Open-Domain Reviews. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9154–9173, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
YASO: A Targeted Sentiment Analysis Evaluation Dataset for Open-Domain Reviews (Orbach et al., EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.721.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.721.mp4
Code
 IBM/yaso-tsa +  additional community code
Data
YASOSST