Refining App Reviews: Dataset, Methodology, and Evaluation

Amrita Singh, Chirag Jain, Mohit Chaudhary, Preethu Rose Anish


Abstract
With the growing number of mobile users, app development has become increasingly lucrative. Reviews on platforms such as Google Play and Apple App Store provide valuable insights to developers, highlighting bugs, suggesting new features, and offering feedback. However, many reviews contain typos, spelling errors, grammar mistakes, and complex sentences, hindering efficient interpretation and slowing down app improvement processes. To tackle this, we introduce RARE (Repository for App review REfinement), a benchmark dataset of 10,000 annotated pairs of original and refined reviews from 10 mobile applications. These reviews were collaboratively refined by humans and large language models (LLMs). We also conducted an evaluation of eight state-of-the-art LLMs for automated review refinement. The top-performing model (Flan-T5) was further used to refine an additional 10,000 reviews, contributing to RARE as a silver corpus.
Anthology ID:
2024.emnlp-industry.44
Volume:
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track
Month:
November
Year:
2024
Address:
Miami, Florida, US
Editors:
Franck Dernoncourt, Daniel Preoţiuc-Pietro, Anastasia Shimorina
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
595–608
Language:
URL:
https://aclanthology.org/2024.emnlp-industry.44
DOI:
Bibkey:
Cite (ACL):
Amrita Singh, Chirag Jain, Mohit Chaudhary, and Preethu Rose Anish. 2024. Refining App Reviews: Dataset, Methodology, and Evaluation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 595–608, Miami, Florida, US. Association for Computational Linguistics.
Cite (Informal):
Refining App Reviews: Dataset, Methodology, and Evaluation (Singh et al., EMNLP 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.emnlp-industry.44.pdf