A Rough Set Formalization of Quantitative Evaluation with Ambiguity

Patrick Paroubek, Xavier Tannier


Abstract
In this paper, we present the founding elements of a formal model of the evaluation paradigm in natural language processing. We propose an abstract model of objective quantitative evaluation based on rough sets, as well as the notion of potential performance space for describing the performance variations corresponding to the ambiguity present in hypothesis data produced by a computer program, when comparing it to the reference data created by humans. A formal model of the evaluation paradigm will be useful for comparing evaluations protocols, investigating evaluation constraint relaxation and getting a better understanding of the evaluation paradigm, provided it is general enough to be able to represent any natural language processing task.
Anthology ID:
L12-1230
Volume:
Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)
Month:
May
Year:
2012
Address:
Istanbul, Turkey
Editors:
Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
2311–2317
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2012/pdf/443_Paper.pdf
DOI:
Bibkey:
Cite (ACL):
Patrick Paroubek and Xavier Tannier. 2012. A Rough Set Formalization of Quantitative Evaluation with Ambiguity. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 2311–2317, Istanbul, Turkey. European Language Resources Association (ELRA).
Cite (Informal):
A Rough Set Formalization of Quantitative Evaluation with Ambiguity (Paroubek & Tannier, LREC 2012)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2012/pdf/443_Paper.pdf