Evaluating Scholarly Impact: Towards Content-Aware Bibliometrics

Saurav Manchanda, George Karypis


Abstract
Quantitatively measuring the impact-related aspects of scientific, engineering, and technological (SET) innovations is a fundamental problem with broad applications. Traditional citation-based measures for assessing the impact of innovations and related entities do not take into account the content of the publications. This limits their ability to provide rigorous quality-related metrics because they cannot account for the reasons that led to a citation. We present approaches to estimate content-aware bibliometrics to quantitatively measure the scholarly impact of a publication. Our approaches assess the impact of a cited publication by the extent to which the cited publication informs the citing publication. We introduce a new metric, called “Content Informed Index” (CII), that uses the content of the paper as a source of distant-supervision, to quantify how much the cited-node informs the citing-node. We evaluate the weights estimated by our approach on three manually annotated datasets, where the annotations quantify the extent of information in the citation. Particularly, we evaluate how well the ranking imposed by our approach associates with the ranking imposed by the manual annotations. CII achieves up to 103% improvement in performance as compared to the second-best performing approach.
Anthology ID:
2021.emnlp-main.488
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Editors:
Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6041–6053
Language:
URL:
https://aclanthology.org/2021.emnlp-main.488
DOI:
10.18653/v1/2021.emnlp-main.488
Bibkey:
Cite (ACL):
Saurav Manchanda and George Karypis. 2021. Evaluating Scholarly Impact: Towards Content-Aware Bibliometrics. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6041–6053, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Evaluating Scholarly Impact: Towards Content-Aware Bibliometrics (Manchanda & Karypis, EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.488.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.488.mp4
Code
 gurdaspuriya/evaluating-scholarly-impact
Data
S2ORC