Abhranil Chandra
2024
VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation
Xuan He
|
Dongfu Jiang
|
Ge Zhang
|
Max Ku
|
Achint Soni
|
Sherman Siu
|
Haonan Chen
|
Abhranil Chandra
|
Ziyan Jiang
|
Aaran Arulraj
|
Kai Wang
|
Quy Duc Do
|
Yuansheng Ni
|
Bohan Lyu
|
Yaswanth Narsupalli
|
Rongqi Fan
|
Zhiheng Lyu
|
Bill Yuchen Lin
|
Wenhu Chen
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
The recent years have witnessed great advances in video generation. However, the development of automatic video metrics is lagging significantly behind. None of the existing metric is able to provide reliable scores over generated videos. The main barrier is the lack of large-scale human-annotated dataset. In this paper, we release VideoFeedback, the first large-scale dataset containing human-provided multi-aspect score over 37.6K synthesized videos from 11 existing video generative models. We train VideoScore (initialized from Mantis)based on VideoFeedback to enable automatic video quality assessment. Experiments show that the Spearman’s correlation betweenVideoScore and humans can reach 77.1 on VideoFeedback-test, beating the prior best metrics by about 50 points. Further result onother held-out EvalCrafter, GenAI-Bench, and VBench show that VideoScore has consistently much higher correlation with humanjudges than other metrics. Due to these results, we believe VideoScore can serve as a great proxy for human raters to (1) rate different video models to track progress (2) simulate fine-grained human feedback in Reinforcement Learning with Human Feedback (RLHF) to improve current video generation models.
You Make me Feel like a Natural Question: Training QA Systems on Transformed Trivia Questions
Tasnim Kabir
|
Yoo Yeon Sung
|
Saptarashmi Bandyopadhyay
|
Hao Zou
|
Abhranil Chandra
|
Jordan Lee Boyd-Graber
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
Training question-answering QA and information retrieval systems for web queries require large, expensive datasets that are difficult to annotate and time-consuming to gather. Moreover, while natural datasets of information-seeking questions are often prone to ambiguity or ill-formed, there are troves of freely available, carefully crafted question datasets for many languages. Thus, we automatically generate shorter, information-seeking questions, resembling web queries in the style of the Natural Questions (NQ) dataset from longer trivia data. Training a QA system on these transformed questions is a viable strategy for alternating to more expensive training setups showing the F1 score difference of less than six points and contrasting the final systems.
Search
Co-authors
- Xuan He 1
- Dongfu Jiang 1
- Ge Zhang 1
- Max Ku 1
- Achint Soni 1
- show all...